Currently not released yet. Beta documentation

Compositor - Multiview - MistProcMultiview

This is a compositor made to serve as an input process. It's in a bit of a special place as it's meant to create a new stream, but only from processing existing streams, using the raw track of these streams and compositing them into one output.

While the input/output are directly compatible with SDI, should you wish to show them in browser or over IP you will need to add an encoder.

In it's current state this tool can be used to:

• Combine/mix multiple streams into one
• Add pictures to a stream
• Layer streams/pictures above each other
• Dynamically alter what is shown
• Sync up multiple outputs to each other
• Both audio and video can be added while it's in use.

tip

• The multiviewer tries to generate a frame at the requested framerate. This means that depending on how sources line up a frame can be behind. In this case the last frame will be used.
• While the multiviewer tries to uphold the requested framerate your hardware will determine whether it is possible. Sometimes frames are duplicated

Configuration

The multiviewer is best configured through the interface. The most notable thing is that every stream added to the Multiview will be put into the given resolution. This means that the stream will need to scale/resize, so setting this correctly is key. Other than that a size and position can be given to each source to customize the positioning. If no position or size has been given a grid will be created with "best fit".

Interface

The multiviewer is quite complex, however if you do it in a few steps it's much easier.

Setting up multiview

First you will need to set up the source to multiview and you will see the designer and additional options appear.

Multiview designer

For adding/removing/compositing we recommend opening the designer, it makes the task much easier.

How to add sources

You can add a source for the multiview to use by adding it as the first value for every row within the multiview. The source can link to either a stream name within MistServer, or a PNG image with RGBA pixelformat within the system. To browse for a picture you can select the browse button to the right of the source field.

Once added you can set up additional rules such as location within the stream, but also how it should behave over the location restraints it has received.

tip

• Any stream that has a JPG output will show the JPG instead of the source name
• PNG needs RGBA pixelformat or it won't load
• The aspect ratio handling and scaling algorithm determine the quality and how the selection is filled. Don't forget to experiment!

Changing the multiview live

If you made edits to the designer, whether it is adding, removing or changing positions of sources you will see the changes appear at the next generated frame.

Configuration file / API

The following would be setting up grid for 9 streams using the scaling defaults

    "multiview": {
        "always_on": true,
        "copyaudio": "0",
        "name": "multiview",
        "processes": [],
        "resolution": "1920x1080",
        "source": "multiview",
        "sources": [
            {
                "h": 360,
                "stream": "example1",
                "text": "custom text",
                "w": 640,
                "x": 0,
                "y": 0
            },
            {
                "h": 360,
                "stream": "example2",
                "text": "",
                "w": 640,
                "x": 640,
                "y": 0
            },
            {
                "h": 360,
                "stream": "example3",
                "text": null,
                "w": 640,
                "x": 1280,
                "y": 0
            },
            {
                "h": 360,
                "stream": "example4",
                "text": null,
                "w": 640,
                "x": 0,
                "y": 360
            },
            {
                "h": 360,
                "stream": "example5",
                "text": null,
                "w": 640,
                "x": 640,
                "y": 360
            },
            {
                "h": 360,
                "stream": "example6",
                "text": null,
                "w": 640,
                "x": 1280,
                "y": 360
            },
            {
                "h": 360,
                "stream": "example7",
                "text": null,
                "w": 640,
                "x": 0,
                "y": 720
            },
            {
                "h": 360,
                "stream": "example8",
                "text": null,
                "w": 640,
                "x": 640,
                "y": 720
            },
            {
                "h": 360,
                "stream": "/path/to/example.png",
                "text": null,
                "w": 640,
                "x": 1280,
                "y": 720
            }
        ],
        "stop_sessions": false,
        "tags": [],
        "target_fps": 60
    }

It's often much easier to set up the multiview using the interface, saving the configuration and then copying it over from the configuration file.

Command line

Using the command line is not recommended for the Multiview as it doesn't allow you to make live changes, it does allow you to create one on the fly.

Example: Setting up a Multiview through commandline

MistProcMultiview --sources {"h":720,"stream":"example1","text":"give","w":960,"x":0,"y":0} --sources {"h":360,"stream":"example2","text":"me","w":480,"x":960,"y":0} --sources {"h":360,"stream":"example3","text":"that","w":480,"x":1440,"y":0} --sources {"h":360,"stream":"example4","text":"sweet","w":480,"x":960,"y":360} --sources {"h":360,"stream":"example5","text":"sdi","w":480,"x":1440,"y":360} --sources {"h":360,"stream":"example6","text":"image","w":480,"x":0,"y":720} --sources {"h":360,"stream":"example7","text":"or","w":480,"x":480,"y":720} --sources {"h":360,"stream":"example8","text":"bust","w":480,"x":960,"y":720} --sources {"h":360,"stream":"example9","text":"!!!","w":480,"x":1440,"y":720} --copyaudio 0 --resolution 1920x1080 --target_fps 60 -s multiview multiview

Required options

Key	Name	Type	Description	Validation
sink	Target stream	string	Stream to which the generated track will be added. Defaults to the source stream. May contain variables.
sources	Source to add	string	source and parameters to add to the multiviewer

Source options

Key	Name	Type	Default / Value	Unit	Help / Description	Options (if applicable)
stream	stream name	String			Stream name within MistServer to use
text	text to display	String	stream name		"" or null for empty, string for a value otherwise stream name will be used.
w	width	Integer	automatic fit	px	width the stream should use
h	height	Integer	automatic fit	px	height the stream should use
x	positioning X	Integer	automatic fit	px	position the stream gets in
y	positioning Y	Integer	automatic fit	px	Position the stream gets in

Optional options

Key	Name	Type	Default	CLI Option	Description	Choices
resolution	Resolution	string	1920x1080	--resolution	Resolution of the output stream (e.g. 1920x1080).
aspect	Aspect ratio handling	select	letterbox	--aspect	How to handle aspect ratio mismatches between input and grid cells.	letterbox – Preserve aspect, add black bars, crop – Preserve aspect, crop excess, stretch – Stretch to fit (may distort image) pattern - repeat pattern to fit the whole selection
scaling	Scaling algorithm	select	integer	--scaling	Algorithm used to scale input sources to fit grid cells.	bilinear – High quality, slower, nearest – Fast, lower quality, integer – Integer-only scaling (fast + HQ)
target_fps	Target framerate	int	0	--target_fps	Target output framerate (fps). If 0, waits for all sources to have new frames before output.	—
copyaudio	Copy audio	string	—	--copyaudio	Copy audio from one of the input sources.	—
printtiming	Print timing	binary	false	--printtiming, -T	Inserts timing information on the sources to use for syncing	true, false
sourcesync	Sync sources	binary	false	--sourcesync, -Y	Attempt to automatically sync sources based on UTC information	true, false

Examples and usages

Due to the very flexible nature of this tool it can be used for multiple purposes. While some purposes are narrowly related they do differ slightly and certain optional parameters become key players depending on the wanted behaviour.

Falback & always active

Falback & Always active

Live streaming has a very obvious limitation that if a stream isn't live, it does not exist. You will need "something" to generate a stream even if your live source is gone. The MistProcMultiview process is the best way to make sure that not only does your stream stay active, but you can replace it with whatever you want to show instead. Whether that is another live stream or simply an picture showing there are technical difficulties. This allows you to keep your stream active to wherever it is being pushed/shown even if your source drops.

Key Settings

• You will require setting up at least 2 sources, The first one is the fallback to drop to and the last one should be the stream to show.
• Always on should be activated
• Your stream sources need to be decoded to UYVY and PCM
• You will need an encoder to H264 and AAC or Opus
• You will need to use free design mode to layer sources over each other.

Free design

• You will want to have the fallback be the first (highest) source
• The source to watch should be below your first source and be layered on top of it.

For this example we have only done a very small video over the fallback picture, we recommend filling the entire stream generally.

Once set up the stream will become available instantly with the stream just showing the PNG:

Then whenever the stream live in this case comes online the stream will add the video and audio of that stream:

Whenever the live stream goes offline or loses connection just the pictures will be shown again, keeping the stream live in the process.

Multiview / Grid

Multiview / Grid

Creating a grid out of multiple streams allows you to create a monitoring stream that can watch multiple processes while not overloading your browser with opening too many streams at the same time. So this is a great method to keep an eye on multiple streams at the same time or give some sort of overview of what else is being shown/streamed.

Key Settings

• The default grid makes things easiest, focus or design are options as well
• Your stream sources need to be decoded to UYVY
• You will need an encoder to H264 to watch the grid back in the browser
• You can optionally name/label streams to keep track of their names or keep notes

This was the first purpose the MistProcMultiview was made for and is the easiest to set up. All you need to do is add several sources, label them and possibly design how the grid should look like.

For adding/removing/compositing we recommend opening the designer, it makes the task much easier. You can even add the stream you are creating, in that case the previous frame will be shown here.

Finally you can watch it in your preferred browser/player:

Syncing & combining streams

Syncing & combining streams

This tackles a big problem in streaming: How do I make sure my streams are synced up? It becomes especially important if you show both sources at the same time and one of the two looks to be behind. In order to make this much easier we've added ways for you to sync up streams. One is automatic that will require showing a changing QR code to both feeds, the other would be manually using timing information shown within the feeds themselves.

Key Settings

• Sync sources needs to be turned on for automatic syncing
• Print timing can help if you're attempting to do things manually, or want to verify the syncing has been done correctly
• A single device needs to show the same QR code to both feeds, the order or timing does not matter.
• Sources will need to be decoded to UYVY and PCM
• You will need an encode to H264 and AAC or Opus for playback

How does syncing sources work

In order to sync sources you will need another device capable of showing a website displaying a QR code. This QR code changes with the UTC time, can be set to a certain frame rate to make it easier for the camera to pick up and should be shown for about 5 seconds. You can find the QR code here

The QR code looks like this:

tip

• The QR code works best at the highest FPS you can get a sharp image with.
• The QR code should never be higher FPS than that of your camera, equal at most

When shown and Sync sources is turned on MistServer will use the information shown on the QR code together with the information we've received from the source to determine the delay to the clock. This delay is also displayed in the print timing that you can activate. Because we can read out the timing of the camera & the timing of what is on screen we can calculate the desync of the camera compared to the clock shown to the camera. This means it's very important to show the QR code from the same device otherwise timing might not end up well. Once we know what kind of delay we're working with we can sync up all the sources to use the timing of the source most delayed to live in order to line them up.

Print timing shows both the draw time and the sync time. Draw time is the time the camera claims it is at. Sync time is the time we try to target based on the other sources and their delays.

Below is a stream created from 3 sources.

• The left source is a camera streaming using RTMP
• The top right source is the same camera sending out using SRT
• The bottom right source is a webcam capturing the QR code

Before the sync you would see that all cameras are close to each other. This isn't a big surprise as they're all streaming to a LAN source. However this can still be much better.

After showing the QR code the streams should quickly sync up. Generally this takes about 5 seconds per source to complete. However should you determine the syncing isn't perfect yet you can show the QR code again at any time to resync all the sources, as long as the sync sources option is activated any timing QR code passing by will be used to update and improve the sync.

tip

• The higher the frame rate the easier syncing becomes as there's more points available to sync the feeds towards
• The fps on the QR code does not need to match the FPS of the camera.
• The fps on the QR code should not exceed the FPS of the camera, that could make the QR code blurry and not take effect.
• Making the QR code bigger does make it easier for MistServer to process & use it.
• If you deactivate sync sources the streams will jump to their most live point, making them desync again.
• You do not need to have print timing in screen in order to work, but it is recommended for manual adjustments

Manually syncing

Manually syncing will allow you to set up the delay per source. This however isn't fully specced out and will be added once it is been added to the Multiviewer.

Watermarking & adding pictures

Watermarking & logos

Watermarking & adding logos can be done quite easily by just overlaying the picture on top of another source. The trick here is that your watermark or picture is the very first source for the Multiviewer. A good thing to know is that if you make any edits to the PNG these will take affect the next frame of the stream, so you can change/update the picture and update the stream live without needing to restart it.

Key settings

• Your watermark/picture needs to be the topmost source
• Your PNG needs to be in RGBA pixel format
• Your other source should be decoded to UYVY and PCM
• You will need to encode to H264 and AAC or Opus for browser playback

Adding a picture

To add a picture you only need to add the path to the PNG as a source & determine the size & location it should get within the stream. This can be done very easily through the designer.

Once added, you most likely need to change the aspect ratio and scaler to determine how the picture should be scaled. In most cases you will want to keep the aspect ratio otherwise the picture will look distorted. Should you update the picture on the actual location that will be changed within the stream as well.

While the scaler will claim it will insert black bars to make sure it fits the frame, these are made invisible should there be something "below" the picture. So our picture only blocks the stream where the actual picture is drawn, any alpha or black bars have been removed from the end result.