Skip to main content

Architecture

Core priciples​

Some basic principles can be seen as recurring themes throughout Mist's architecture. These are:

  • Modularity
  • Compatibility
  • Resilience
  • Reusability

Since these themes are fairly related to each other, sometimes they tend to blur together a bit.

One binary, one task (reusability, modularity, resilience)​

This decision shows immediately when you look at MistServer: it contains a lot of binaries with Mist* naming. This is because MistServer tries to follow the Unix principle of one binary for one task:

  • Each stream's data on the system is maintained by a MistIn* input binary.
  • Each outgoing connection is handled by a MistOut* output binary.
  • Modifications on stream data are handled by MistProc* processing binaries.
  • Each session is represented by a running MistSession binary.
  • The MistController binary monitors and controls all of the above, and provides the MistServer API as a single point of control.
  • Separately, several MistAnalyser* and MistUtil* binaries provide developing and debugging support without being a part of the "core" MistServer binaries as such. (An installed instance does not use these binaries.)

This very fragmented setup, where each task is handled by not just a separate thread but a complete separate process, was chosen for several reasons. There is of course some overhead associated with this approach, and especially when accepting new connections this overhead is noticable (this is why high-throughput webservers generally do not spawn a process for every connection). However, since MistServer is a media server the positives by far outweigh the negatives:

  • Processes and threads are almost the same thing in Linux, and the kernel is very good about re-using static memory allocations, so the (extra) overhead of running another process is negligible.
  • Media connections tend to be long-lived and high in bandwidth use. (As opposed to most websites, which have short-lived low-bandwidth connections instead.) This means network capacity usually runs out long before CPU or RAM capacity runs out, even with the extra overhead.
  • Having a process per connection means a single connection can at most crash the handler for their own connection, rather than the whole system at once.
  • Having a process per connection means we can replace the listening processes during an update, but keep the already-connected processes running on the old version, allowing for rolling updates without any downtime.

Code structure (reusability, modularity)​

The code for MistServer is split in several ways as well:

A collection of more general media-related libraries is present in the lib directory, together called libmist. Each of these smaller libraries is made to help with one particular task or protocol or format, and attempts to avoid reliance on other libraries where possible to encourage their re-use. They are largely named after the "thing" they do or help with, and adding new libraries without this kind of clear naming is discouraged.

Inputs, outputs, analysers and processes all have their own directories and are --- again --- named after the "thing" they do. They have only a .cpp and a .h file each, and generally have no other files specific to them in those directories. Most of these do have a single generic file that applies to all, containing the main function and most of the boilerplate setup code.

Decentralized processing (resilience, reusability)​

Flexible inputs, strict outputs (compatibility)​

The flow of control is rather loosely defined on purpose: "inputs" are started as-needed, directly by the "output" that wants to receive media data. The outputs are spawned from listening processes that wait for connections on sockets, and these listeners are started by the controller. Each output as well as live inputs report statistics and health data to the controller.

The media data itself is made available in DTSC format through shared memory pages, which can freely be read by all processes of the same system user. Metadata on the media data is provided through a custom binary structure, which is written to only by the input and read from by the outputs. This structure is locked only while being written to (roughly once per second), and the many readers do not have to lock the structure to read simultaneously. A similar method is used to report back to the controller.

Starting order​

The below graph shows which binaries start which other binaries within MistServer:

Inter-process communication​

The below graph shows which binaries communicate with other binaries within MistServer:

Error recovery​

Now, this becomes especially interesting when any of MistServer's processes crash or fail in some other way. Since MistServer was written with dependability in mind, you may never have experienced this. So let me walk you through what happens if something goes wrong:

Should any of the outputs crash or fail, the single connection it was maintaining will be severed. Nobody else will be affected, and the controller knows that and when the process has crashed because it will stop to report back at regular intervals. If the process is frozen or stuck in some kind of loop, the controller will forcibly kill the process to ensure the stability of the rest of the system. There is no data to clean up, since all data used by outputs is in shared structures maintained by the corresponding input.

Should any of the inputs crash or fail, each input has a dedicated "angel process" watching over it that will take notice. Since inputs maintain the shared memory structure, there is a potential to leak a lot of memory should these processes suddenly disappear without doing proper cleanup. The angel process will clean up all memory left behind by the input, and then re-start the input. The outputs never even notice the input has stopped and restarted, and will just take slightly longer to load while the input is recovered. No connections are severed at all in this case (unless the stream in question was a live stream; since the timing information is lost during cleanup).

Should the controller itself crash, this too has an angel process watching over it for the same reason that the inputs do. The controller maintains several structures that contain state information, as well as the structures that the inputs and outputs use to report back. These structures are all known beforehand, which lets us do a neat trick: instead of cleaning up the structures, the newly started replacement controller loads its state information from the existing structures. This allows the controller to literally pick up where the previous one left off, without any of the inputs or outputs even noticing what happened.

Especially cool is that the above behaviour also allows for rolling updates. The MistServer binaries can be replaced by new versions, and the controller told to restart itself. Any new connections will use the newly installed binaries while old connections keep using the old ones. The same holds true for the input processes. Eventually, the whole server will be updated, without ever dropping a single connection in the process.

DTSC format​

DTSC stands for "DDVTech Stream Container", and is the format MistServer internally converts all inputs into. The outputs then read this format and convert it to whatever they are supposed to output to the end-user. Doing things this way, allows us to generically write outputs and inputs without needing to know which of each will be available in advance. It's one of the biggest reasons why MistServer is so modular, and why we have been able to add inputs and outputs so regularly.

DTSC itself is a container/transport format taking inspiration from the FLV, JSON, MKV and MP4 formats. It tries to take the good parts of those formats without making anything overly complicated. It's packet-based, with a (repeatable) header that contains initialization and seeking information. All packets are size-prepended, and the data format is based on a simplified form of binary JSON. These properties allow DTSC to be both used as a storage format on disk and as a transport format over networks.

The full DTSC specification is available here.

Besides the internal use (which only usually exists in RAM and is never written to disk at any point), DTSC is used by MistServer in two more places: our header files (the .dtsh files you may have noticed appearing alongside your VoD assets) and by the DTSC Pull input (for cross-server replication).

When MistServer's various inputs read a VoD file from storage, they generate a DTSC header and store it beside the original file as a .dtsh file. On future accesses to the file, this header is used to speed up loading from the file. It can safely be deleted (and will regenerate the next time the file is accessed) and will auto-delete if the format changes because of MistServer updates. This file helps us provide a consistent speed across all media storage formats, and provides an index for files that normally do not have an index (such as TS files) or where such an index is optional.

The DTSC Pull input allows you to pull live streams from other MistServer instances, using DTSC as the transport. This means it is a live replication and distribution format that is compatible with an unlimited number of tracks/qualities and works for all codecs. Unfortunately, MistServer is the only software (at time of writing) that has implemented DTSC as a streaming input/output format, so you can only take advantage of DTSC distribution between MistServer instances (for example, for load balancing live streams). There are plans to also make DTSC usable for VoD distribution in the near future.