Skip to main content

Generic bidirectional WebSocket-based streaming transport specification

Various formats that MistServer supports are available over a bidirectional stream connection (e.g. WebSockets, but this also currently covers WebRTC data channels and in the future will cover WebTransport). When there is a bidirectional connection like that, it is possible to control the playback of a progressively delivered stream of media data with command messages. The MistServer Stream Playback Control Protocol describes these command messages and what they mean.

Goals of this specification

  • A simple format that is nearly trivial to implement
  • Flexible enough to transport any codec, past, current or future
  • Framing is provided by the transport protocol
  • Streams track data in either direction, or both at once
  • Compatible with prepackaged / non-raw codec data as well

Basic packet types

If media data and command messages are combined on a single transport channel, you can tell them apart because control messages are always JSON objects, and as such must start with a { character. Messages that start with anything else shall not be considered control messages and may not be processed as such.

  • Command message packets contain signalling messages in JSON format, always start with a { character
  • Binary packets contain media data, with a 12-byte header prepended

Command messages​

JSON data in plain text UTF-8 format. May be ignored if not an object (always starts with a { character). May be ignored if there is no type member. type member is a string listing the message type. Unrecognized types should be (silently) ignored.

In addition, an optional data member may contain more information as described below for each command. "Server" commands are transmitted by the media server to the client. "Client" commands are transmitted by the client to the media server.

Server command type: on_stop​

Transmitted when playback stops.

Data members:

  • current: Timestamp of the next-to-be-sent media packet
  • begin: Lowest playable timestamp (begin of media buffer)
  • end: Highest playable timestamp (end of media buffer)

Server command type: set_speed​

Transmitted when playback speed changes.

Data members:

  • play_rate_prev: Previous playback speed (may be "auto", "fast-forward", or a float multiplier)
  • play_rate_curr: New playback speed (same possible values as play_rate_prev)
  • live_point: Optional field. Set to true if playback will attempt to stay at the live point.
  • reason: Optional field. A text string describing the reason the speed changed (may be "at_live_point")

Server command type: pause​

Transmitted when playback (un)pauses.

Data members:

  • begin: Lowest playable timestamp (begin of media buffer)
  • end: Highest playable timestamp (end of media buffer)
  • reason: Optional field. A text string describing the reason the pause occurred (may be "at_dead_point")

Besides the data member, there is also a top-level member paused which is set to either true or false to reflect the current state.

Server command type: on_time​

Transmitted at regular intervals (approximately once per second during playback, more often around changes in playback speed/state).

Data members:

  • current: Current playback position timestamp
  • next: Timestamp of the next-to-be-sent media packet
  • begin: Lowest playable timestamp (begin of media buffer)
  • end: Highest playable timestamp (end of media buffer)
  • play_rate_curr: Current playback speed (may be "auto", "fast-forward", or a float multiplier)
  • tracks: Array of numeric track indexes
  • jitter: Ingest jitter for the current stream in milliseconds (recommended distance to start playback away from the live point, for live streams)
  • unixoffset: Optional for VoD streams. Always present for live streams. The number that must be added to stream timestamps to get the unix time in milliseconds that the timestamp corresponds to.

Server command type: warning​

Transmitted when the server wants to warn the client about something that isn't a critical stop condition. Contains a single warning member with a human-readable text string inside.

Client command type: seek​

Requests that playback position changes. All data members are optional, but either seek_time or unix must be present in the data object.

Data members:

  • seek_time: Millisecond position in the stream timeline to continue playback at. May also be the string "live" to indicate the live-most playback point.
  • unix: Unix time in milliseconds indicating the position in the stream timeline to continue playback at.
  • ff_to: After seek, enable fast-forward mode until the given stream timestamp.
  • ff_add: After seek, enable fast-forward mode for the given number of milliseconds.

Client command type: pause​

Request to pause or unpause playback (toggles between the states). No further payload. Server will follow up with a server pause command.

Client command type: hold​

Request to pause playback (does not toggle between states). No further payload. Server will follow up with a server pause command.

Client command type: stop​

Request to stop playback. No further payload. On receipt, the server will disconnect.

Client command type: tracks​

Request to change the track selection. May contain the following optional members:

  • video: Video track selector
  • audio: Audio track selector
  • meta: Metadata track selector
  • seek_time: Stream timeline timestamp to simultaneously seek towards.

Client command type: fast_forward​

Requests that the playback speed be set to "fast-forward" for some time.

  • ff_to: After seek, enable fast-forward mode until the given stream timestamp.
  • ff_add: After seek, enable fast-forward mode for the given number of milliseconds.

Client command type: set_speed​

Request to change playback rate. Must contain a single member play_rate which may be set to the string auto, the string fast-forward or a float playback speed multiplier (1.0 = real-time, 0.5 = half-time, etc). Server will respond with a set_speed server command message.

Client command type: play​

Resume or start playback.

Optionally may have seek_time member, which indicates the wanted starting playback position. If omitted, playback should either resume from where it was last paused, or if no playback occurred yet at the beginning for VoD / the live point for live.

Optionally may have a ff_add member, which adds a request to fast-forward for the given number of milliseconds of media data.

Protocol-specific extensions and information​

Payload format: MP4 data (*.mp4 paths)​

MP4 data consists of an MP4 mux suitable to be fed into a MSE context.

Client command type: request_codec_data​

Request for track index to codec mapping. Optionally contains supported_codecs member containing an array of strings that list the supported codecs (if not, supported codec list is guessed/assumed).

Server command type: codec_data​

Response to request_codec_data, may also be sent at any time the other end considers useful (e.g. available track list changing, initial connection). data.current contains current playback time, if available. data.codecs contains an array of codec name strings. data.tracks contains an array of corresponding track indexes.

Payload format: EBML data (*.mkv, *.webm paths)​

EBML data consists of an EBML mux suitable to be fed into a MSE context.

This format has not yet been finalized, but is planned to be very similar to the MP4 format in behaviour.

Payload format: Raw data (*.raw, *.h264 paths)​

The raw codec data over WebSockets protocol mixes control and media over a single connection.

Media data is formatted as a 12-byte statically sized header, followed by the raw codec data inside the same protocol-level packet/message.

Header format:

  • 1 byte track index (0-indexed)
  • 1 byte frame type (0 = regular, 1 = keyframe, 2 = init data)
  • 8 bytes timestamp in milliseconds (network byte order == little endian)
  • 2 bytes time DTS/PTS offset in milliseconds (network byte order == little endian)

The header is followed by rest-of-the-packet bytes of raw codec data. The timestamp and offset should be ignored when receiving type 2 packets and should be zeroed out when sending them. The payload for type 2 packets after the header for all codecs is the full contents of libav's codec_private field (which is, incidentally, also identical to CodecPrivate elements in WebM/Matroska format).

Binary format is identical to the WebCodec registration for the codec in use, where "init data" is the DecoderConfig. For codecs that have no WebCodec registration, ffmpeg's codec_private_data can be used for init data and ffmpeg's encoded frame data format for the other packets.

Client command type: request_codec_data​

In addition to the generic commands, WS/RAW also supports the command "request_codec_data", which may include an optional "supported_codecs" member containing an array of strings (listing the supported codecs) or an optional "supported_combinations" member which contains the supported codecs in a more complex format that follows the same specification as MistServer's output "codecs" field.

If sent, the server will reply with a "codec_data" command message. In its "data" member, it contains "current" (current playback timestamp), "codecs" (list of codecs, each entry follows either the WebCodecs codec naming spec or is a Mist-internal codec name preceded by an exclamation point character if we could not generate the appropriate codec name) and "tracks" (array of track indexes, in the same order as the codecs array).

Server command type: info​

Right before sending packets with codec initialization data, the server will transmit an info type message, containing a msg member in its data member with as value Sending header and a tracks member inside its data member that contains an array of the track indexes that will be transmitted from here on forward. This message (and the initialization data) will be re-transmitted every time the transmitted track selection changes, before any (new) media data is transmitted.