The MistServer Load balancer

The MistServer load balancer has been in beta status for several years, while also running in production at most of our high-end users. The core of the load balancer is complete, functional and stable, however we have many improvements we want to add and the user experience isn't great (to put it lightly). Therefore we never considered it done or ready for general release. It is however fully capable of monitoring a complete CDN worth of MistServers, directing viewers and contributors to the correct nodes and making sure streams are spread evenly over the network. The current version of the load balancer can keep track of the state and location of each server and use this information to calculate estimated usage of future viewers and distribute them accordingly over the available servers.

Compiling the Load Balancer

The best way to get the Load balancer is to simply compile it yourself. By adding -DLOAD_BALANCE=true to your compile flags you'll be enabling the load balancer and will see a new binary: MistUtilLoad.

Load balancer usage instructions

The load balancer is the MistUtilLoad application among the MistServer binaries. MistUtilLoad needs to be started with as arguments the servers to balance and a single prometheus passphrase that is set identically on all the servers themselves as well (this setting can be changed on the overview page of MistServer's web interface). This is because the balancer pings the prometheus endpoints of the servers to get real-time statistics.

The servers are given using the -s IP_or_hostname[:PORT] argument, repeated for each server (the port should be the API port, and defaults to 4242 if not given). For example: -s 1.2.3.4 will add 1.2.3.4 with API port 4242 to the server list -s 1.2.3.4:4444 will add 1.2.3.4 with API port 4444 to the server list Use the option as many times as you have servers. It's not required to use IP addresses either - hostnames work just as well. Do note that

The prometheus passphrase must be given using the -P option: -P PASSPHRASE

The -p option sets the port number for handling balancing requests. The default is 8042.

Finally, the -g or --debug (these both work) argument can be used for more debug output:

-g 3 is warnings/errors only (default)
-g 4 will turn on verbose mode
-g 5 will turn on extremely verbose mode

For other commandline arguments, see the --help output for more details. Especially setting a fallback server could be handy.

An example of booting MistUtilLoad would be:

MistUtilLoad -P PASSPHRASE -s 123.123.123.123:4242 -s 234.234.234.234:4242 -s 123.234.123.234:4242

This starts MistUtilLoad with 3 servers added, located at 123.123.123.123, 234.234.234.234 and 123.234.123.234, using PASSPHRASE as passphrase.

Systemd service script

The most convenient method to run the MistServer loadbalancer would be through a systemd service script. For example:

[Unit]
Description=MistServer loadbalancer
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/MistUtilLoad -g 4 -P PASSPHRASE -s http://123.123.123.123:4242 -s http://234.234.234.234:4242 -s htpp://123.234.123.234:4242
Restart=always
RestartSec=2
TasksMax=infinity
TimeoutStopSec=8

[Install]
WantedBy=multi-user.target

You'll want to change the ExecStart= line to point to where you keep your MistServer binaries and to point to the correct server list. Adding/removing servers can be done by editing the script and restarting the service, but you can also adjust the configuration in real time through live control.

Live control

Once running, MistUtilLoad makes available the following URLs:

/STREAMNAME           will give you the best server to contact for that stream.
/STREAMNAME?proto=A   will give a HTTP redirect response for protocol A on the best server for the given stream.
/                     will give you stats for all servers.
/?host=A              will give you stats for server A
/?stream=A            will give you the total current viewer count for stream A over all servers
/?source=A            will give the source stream to be used to pull stream A from another server (used internally by the load-balanced stream source)
/?viewers=1           will give totals for all servers added together, per stream
/?weights={JSON}      will give the current weights and settings for the load balancing algorithm. Providing new values (in the same format) as the JSON parameter will change the current settings immediately.
/?lstserver=1         will give you a list of servers currently being balanced
/?addserver=A         will add the given server to the balance list. Same format as the '-s' command line option
/?delserver=A         will stop balancing the given server. Again, same format as '-s'.

The / URL (both with and without params) is only accessible from the local machine if running with the --localmode command line option. This is recommended, since otherwise all addresses can access it without any authentication. Keep this in mind when setting up your network security (firewall appropriately!).

For example, if MistUtilLoad is running at 123.123.123.123:8042 getting the best server for the stream livestream01 would be: http://123.123.123.123:8042/livestream01

Now one of two things can happen. Either you get the server IP/host that should be used as a response, and you can use that directly in any of your scripts to send your viewer to the correct server. Or, you get FULL (or your fallback set with -F FALLBACK_HERE). If you get a FULL response that means either there is no server available with enough resources to handle the request, or the stream cannot be found on any server. We recommend handling that last case by manually sending to a random server, so the load of these failed requests is still spread over all the servers to some degree. You could also stop the view attempt then and there, but that means your system goes down if the load balancer goes down. In general you'd want to prevent any single point of failure.

If using the ?proto= format, the reply will be an error message or a HTTP 3XX redirect with a Location header set to the final URL for the given protocol on the recommended server. If the protocol is set to HTTP, that will be a html page with protocol autodetection, otherwise it will be a direct link to the playback for the requested protocol.

Geo-IP balancing

Both formats optionally allow you to use Geo-IP-based balancing. To use this, the user's latitude and longitude must be provided to the balancer (server locations should be set in the configuration of each server, and are automatically forwarded to the balancer and update in real-time when changed in the config). These coordinates must be in decimal degree format, and can be either provided through the GET parameters ?lat=X&lon=Y or the custom HTTP request headers X-Latitude and X-Longitude. If no coordinates are given (or both are zero), the system ignores coordinates completely when balancing. Geo-based balancing uses a weight setting, just like all other balancing options (CPU, RAM, bandwidth) and the weights all come with a sensible default setting.

We do not offer any Geo-IP databases, so if you want to use this feature you'll have to provide one yourself or through a third party. As long as you can get (approximate) coordinates in the correct format, it really doesn't matter which database you use - but we have good experiences with MaxMind Geolite, which is available for free.

Using the Load balancer together with MistServer

The load balancer's main feature of course is directing viewers to the best server instance to connect to, but there is a second type of balancing that is equally useful: "input" balancing. This is the balancing used by MistServer instances to find which other instance already has a stream active and then pull it from that instance. Alternatively, you can have each instance auto-push every received stream to every instance, but this is of course wasteful of bandwidth if there is no need to send that stream to every node (because the stream is not that high in demand, for example). Using this method also prevents ingesting a different-content-but-identically-named stream on multiple nodes, which can be very confusing if it happens.

Each stream within MistServer can be instructed to check with the load balancer by setting the source to balance:http://balancer_host:port. This will tell MistServer that if the stream is inactive it should talk to the load balancer & request where to pull the stream from. A fallback can be set by appending ?fallback=URL to the source address, this will be used if the load balancer is not responding, or is unable to provide a server to pull from (e.g. the stream is not active yet). A common style is to set the fallback to push:// to be able to accept incoming pushes (and thus become origin server for that stream) if the stream is not active in the network yet.

Traditional CDN: Source & Edge nodes

A traditional CDN can be created by having any of the source nodes accept live streams. Whether your source streams contact the load balancer is optional, but can be done by using a Hybrid source setup. The idea here is that there's a pool of one or multiple source servers that contain the streams to be served to viewers. Edge nodes are set up to pull from source (or even other Edge) nodes & provide streams to viewers. Whether viewers can view directly from source servers is up to you, traditionally only edge servers provide the stream to viewers but MistServer also supports a hybrid approach where every server is both edge and origin simultaneously.

Any edge server should be configured to pull through the load balancer: To use this feature, the balancer needs to be accessible for your servers (i.e. not firewalled), and the servers need to be configured to ask it where to get a stream through the source setting. The source setting can be configured for example like this: balance:http://balancer_host:8042/?fallback=dtsc://server/streamname

balance: is the key word that tells Mist to use load balancer logic.
http://balancer_host:8042/ is the address and port where the load balancer can be reached by the server.
?fallback=dtsc://server/streamname is the source to use instead, if the balancer cannot be reached or the balancer doesn't have any known servers with this stream available. This can be any source type that works in normal operation modes as well. In this case we've set it to the "main" origin server as a fallback.

Hybrid source setup

In a hybrid set up every server can be either a source or an edge server for every stream within the network. Essentially every server is or can be both and is fully available to the network in terms of capacity. While this does mean your sources can run into issues should you saturate your capacity you can run the entire CDN much cheaper than the traditional method as you would require less servers, less total bandwidth, and you'll have less unused capacity.

An easy method of telling Mist instances to first try and pull from the network and if not available within the network accept the stream as incoming push instead is to set up the source as follows: balance:http://balancer_host:8042/?fallback=push://

balance: is the key word that tells Mist to use load balancer logic.
http://balancer_host:8042/ is the address and port where the load balancer can be reached by the server.
?fallback=push:// is the source to use instead, if the balancer cannot be reached or the balancer doesn't have any known servers with this stream available. With push:// that means accept the push instead. If the stream is already active pushes are rejected and the stream will be pulled off the node that has the stream active.

Instead of using a fallback, using the Streamkey push method would also work. This bypasses the check at the load balancer should a stream not be active & allow for a push in, after which the stream becomes active and all other streams can start pulling in the stream as well.

Push method

The push method does use more bandwidth than the pull method, however there are a few upsides to it as well. It's much easier to understand how streams are shared within your network and you can safely assume any stream is always available at any time on every server instead of loaded in when necessary. While all you need to do is set up a push towards all other servers we generally recommend setting up the push to be DTSC. The reason for this is that any stream coming in using DTSC would be automatically tagged with #replicated & is forbidden to be used by any automatic push unless specifically instructed to ignore the #replicated tag.

To push towards another server using DTSC use the following syntax:

dtsc://serveraddress:DTSCPORT/streamname

Which method is recommended?

Unless you have very specific limitations or requirements we would always recommend the pull method for a few reasons:

Faster to scale: The only thing you need to do is add the new server to the load balancer & copy over the "default" configuration. You've got a server setup in a minute.
Better on bandwidth: A server in pull mode only receives the stream when there's viewers. So there's less bandwidth wasted.
Pull mode allows the full buffer to be passed along, not just the live edge. This is mostly handy for segmented protocols such as HLS.