Streaming protocols

What should you know about streaming

A protocol describes the way computers can communicate with each other. This is used in streaming media to create a connection between servers and viewers. Media itself is in a container format, which describes how media is packaged. Then Within those packages are Codecs which actually describe how the media can be used. The streaming methods described below are combinations of these three that will be able to fit in every type of media delivery.

Streaming Methods

To keep things as basic as can be there's 3 methods of handling media data that you want to remember and almost every method of delivering media will fit in these categories.

Real time streaming

Stream based protocols are true streaming protocols. It has two-way communication between the server and the client and maintain an open connection. This allows for faster, more efficient delivery and lower latency. These properties make it the best option, if available. On the downside you need a player that supports the protocol, which often makes it unavailable for browsers.

Pseudo streaming

This is a method where you have a HTTP(S) server and you pretend all streams are files. However it's not always a file, it could actually be a live stream and you're just sending it pretending there’s a file. You lose out on latency, but gain a lot of compatibility as most players will be able to use this as long as the container format is supported.

Segmented streaming

This is the current most popular method. Here you cut up a stream in several parts and you have an index that shows how to reach every part. That index updates when new parts become available. As long as the player knows how to make sense of the index it should be able to play the stream and an advantage is that due to the nature of how it works trick play options (seeking, reverse playback, fast forward, etc.) is easier to do.

Protocol, connection type and Container format are a little blurry

This is something to keep in mind, Protocols, Connection type and Container format are related and influence each other. Certain combinations are not allowed, or required. Since this is an introductionary article we'll simplify things a little bit, in reality things are not so black and white.

Transport types

Then for the transport types you also have several options. The two big ones that everything is based on are:

• User Datagram Protocol (UDP)
• Transmission Control Protocol (TCP)

Then we have the transport types that build onto TCP and UDP:

• Hyper Text Transport Protocol (HTTP(S))
• Secure Reliable Transport (SRT)
• Reliable Internet Stream Transport (RIST)

Note: "Protocol" in the cases above does not refer to a protocol in the same context as discussed in the rest of the topic. UDP, TCP and HTTP are transport protocols, while in most of the article we're talking about streaming protocols.

What you need to know about these transport types:

UDP

UDP is a type of connection where one side is listening and anyone can send towards it. There's no form of handshake or control, what you receive is what you get. Using this method packets tend to go missing or appear in the wrong order for all intends and purposes this is very much a fire and forget connection. The big benefit is that it's fast, but will be unreliable if your connection isn't good.

SRT, RIST

These are UDP based transports. They attempt to "fix" the packet loss and packet reordering issues that UDP normally has in various ways.

TCP

TCP is a connection where both sides are aware of each other. First a handshake is made and both sides know exactly what is send, needs to be send and what the order is. This is a slower type of connection as almost everything gets double-checked, but you're guaranteed that everything comes in as you want.

HTTP

HTTP is the transport most commonly known for serving webpages. It has gone through several modernisation updates, currently the latest version is HTTP3, but HTTP1.1 is still widely used. HTTP versions under 3 use TCP as their underlying transport, while HTTP3 uses UDP as its underlying transport. This means the exact behaviour and pros and cons are dependant on the version and specific device in use.

Summing up the methods and connection types:

Streaming Method	Latency	Trick play	Player support
Real time streaming	Low	Yes	Custom players
Pseudo streaming	Medium	No	Native playback
Segmented streaming	High	Yes	Native playback

Connection type	Latency	packet loss	Reliable
UDP	Low	Yes	Only in good condition
TCP	Medium	No	In almost all conditions

So what should I use?

This is the obvious question to ask, and really the answer is any of these are a good option. You should pick the method that best fits your or your users needs.

Ultra low latency

The best latency will always be a Real time streaming over UDP combination. There will always be two challenges with this combination however:

• Bad connections will be problematic
• A custom player is almost always necessary

Therefore we recommend having a Pseudo streaming or Segmented streaming as option to fall back towards.

Reliability

For reliability you mostly want to look at options usingTCP. That's not to say that UDP cannot be reliable, it's just easier to achieve when using TCP. This is especially true when you're talking about how to reach your end users. When we are talking between server traffic you can generally assume the connection is good enough to use UDP, which allows you to share media quickly between servers and do TCP for the part of the connection where the public internet tends to become more unreliable.

Where the protocols that MistServer supports line up

Common name	Streaming method	Transport type	Container	Where	Expected Latency
RTMP	Real time streaming	TCP	Flash Video	To servers	< 3s
WebRTC	Real time streaming	UDP	RTP	Browsers	< 1s
RTSP	Real time streaming	UDP and/or TCP	RTP	Custom players, servers	< 1s
SRT	Real time streaming	SRT	MPEG2-TS	to servers	< 3s
RIST	Real time streaming	RIST	MPEG2-TS	to servers	< 3s
MPEG-TS	It's complicated	It's complicated	MPEG2-TS	It's complicated	It's complicated
MP4 (MPEG4)	Pseudo streaming	HTTP	MP4	Browsers, players	< 5s
WS/MP4	Pseudo streaming	HTTP	MP4	Browsers	< 3s
MKV / WebM / EBML	Pseudo streaming	HTTP	Matroska	Storage, Edge, players	< 5s
FLV	Pseudo streaming	HTTP	Flash Video	Legacy	< 5s
HLS	Segmented streaming	HTTP	MPEG2-TS / CMAF	Apple, scripted players	< 30s
LL-HLS	Segmented streaming	HTTP	CMAF	Apple, scripted players	< 5s
HSS	Segmented streaming	HTTP	CMAF	Legacy	< 30s
DASH	Segmented streaming	HTTP	MPEG2-TS / CMAF	Browsers, non-Apple	< 30s
HDS	Segmented streaming	HTTP	Flash Video	Legacy	< 30s
MP3	Pseudo streaming	HTTP	MP3	Browsers, Players	< 5s