Viblast Backend: Installation and Configuration

(Draft)

Overview

Viblast's technology adds a peer-to-peer layer on top of existing video streaming infrastructure transforming it into a peer-assisted CDN - Viblast PDN.

Each Viblast peer (viewer) connects to the Viblast backend using the Websocket protocol. This connection is used for:

• introducing Viblast peers to one another and helping them establish WebRTC data channel connections
• reporting metrics data from Viblast peers - traffic data, QoE data, etc.

It is important to note that the connection of a Viblast peer to the Viblast backend is not used for transferring any video data. All video data is transferred either from the CDN or between peers using WebRTC data channel connections.

Video playback does not depend on the Viblast backend. If some Viblast peers cannot establish a connection to the Viblast backend they will still be able to play video using the CDN connection.

Viblast Backend Structure

The Viblast backend consists of several applications - Viblast Tracker, Viblast Aggregator and Viblast Portal.

Viblast Tracker

Each Viblast peer (viewer) connects to Viblast tracker using the Websocket protocol and keeps this connection while consuming the video stream. Viblast tracker introduces peers to one another and helps them establish peer-to-peer connections. Peers regularly announce to the Tracker which video segment they are currently playing, the amount of data they have downloaded from the CDN and from other peers, how much data they have uploaded to other peers, QoE data, etc. As a response, the peer receives a set of other peers that are suitable for establishing peer-to-peer connections.

Viblast Aggregator

The purpose of Viblast Aggregator is to aggregate data reported by all peers (viewers) into usable and easy to comprehend metrics. Each metric is aggregated as time-series data with 1min resolution. Common metrics collected and stored by the Viblast Aggregator are:

• Number of viewers of a given stream
• Number of viewers of a given stream by quality
• Amount of data downloaded from the peer-to-peer network (PDN)
• Amount of data downloaded from the CDN
• Browser usage of a stream's viewers
• Browser usage of a stream's viewers by quality

Metrics time series data is stored into the Graphite monitoring tool (https://github.com/graphite-project).

Viblast Portal

Viblast Portal helps create and manage streams and visualize metrics data collected by Viblast Aggregator.

Viblast Backend Deployment

The Viblast backend is deployed on Linux servers. All examples that follow assume usage of Ubuntu 14.04LTS. Support for other distributions is available on request.

Viblast Tracker and Viblast Aggregator are distributed as Erlang applications. Viblast Portal is a Python/Django application. Additionally, the Viblast backend requires MySQL server and Graphite monitoring tool.

The Viblast backend supports different deployment architectures.

Single-Node Deployment

For not very large deployments of up to ~20,000 viewers the recommended deployment is single-node deployment.

Multi-Node Deployment without Clustering

For deployments where more than 50,000 concurrent viewers should be handled, the Viblast backend supports multi-node deployments. The Viblast backend can be deployed on multiple independent nodes and load-balanced using HAProxy or another load balancer.

Deploying Viblast backend without clustering support makes the deployment simpler, but has some shortcomings that are discussed below.

Multi-Node Deployment with a Single Aggregator

Viblast Aggregator can be connected to multiple tracker nodes and collect data from all of them. This is the simplest multi-node deployment architecture, but obviously it does not scale well. We do not recommend to connect one Viblast Aggregator to more than three tracker nodes.

Multi-Node Deployment with Independent Aggregators

When single-aggregator deployment is insufficient, deployment with multiple aggregators is available. Again, not using clustering support in Viblast's backend simplifies the deployment architecture, but comes with some shortcomings.

In this particular case, the main disadvantage is that each Viblast Aggregator produces independent data that should be additionally aggregated.

Multi-Node Cluster Deployment

Viblast backend supports building clusters of Viblast tracker nodes and of Viblast Aggregator nodes. This deployment is recommended for production loads above 100K concurrent viewers.

The Viblast backend clustering supports runtime cluster configuration changes such as adding nodes, inspecting the state of the nodes, replacing nodes, etc.

Common Settings of the Viblast backend

Create Viblast user

The Viblast backend should be run by an unprivileged user. It is common to create a viblast user and run all backend application as that user.

To create the viblast user:

sudo adduser --disabled-password --gecos '' viblast

Control System Limits

Viblast Tracker installations running production workloads may need system limits and kernel parameters tuning in order to handle a production-suitable number of concurrent connections. The main setting that needs adjustment is the max number of open files, also known as ulimit -n. The default value on many operating systems is too low for Viblast Tracker (e.g. 1024 on several Linux distributions).

To increase the limit of the maximum number of open files, edit the /etc/security/limits.conf and change the lines that specify nofile limits or add them if they are not set:

*               soft    nofile          1048576
*               hard    nofile          1048576
root            soft    nofile          1048576
root            hard    nofile          1048576

Additionally, we need to tune Linux kernel networking parameters to support a large number of simultaneous connections:

net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_mem=50576   64768   98152
net.core.netdev_max_backlog=2500

Viblast Tracker Deployment

The Viblast Tracker application is distributed as generic Linux tar package vbt.tar.gz. The current build supports Ubuntu Linux 14.04 LTS. Builds for other Linux distributions are available on request.

Viblast Tracker Node

To install, untar the package into a directory (for example, in /opt). The created directory structure is:

vbt
├── bin
├── data
├── erts-6.4
├── etc
├── lib
├── log
└── releases

Tracker configuration is stored in vbt/etc directory. The scripts for starting/stopping viblast Tracker are found in vbt/bin directory. By default, the logs are stored in vbt/log directory and vbt/data contains some runtime data about the state of the Viblast Tracker cluster. The location of these directories could be configured.

Viblast Tracker Configuration Files

Viblast tracker is configured using the following files:

vbt/etc/
├── vm.args
└── app.config

vm.args

A fragment of vm.args with default values is shown below:

## Name of the node
-name [email protected]

## Cookie for distributed Erlang
-setcookie vbt

These parameters are used when connecting Tracker and Aggregator nodes or when building Tracker cluster. If you plan to connect Viblast Aggregator or other Tracker nodes to this node, you have to change these parameters.

The first parameter is node name. If the private IP address of the server is 172.16.0.17, then a suitable node name would be [email protected].

The second parameter is setcookie parameter. This parameter is a shared secret parameter. All nodes that should be connected to one another have to share the same secret. A suitable value for this parameter is some secret hash. In our example, we will set this to 74b87337454200d4d33f80c4663dc5e5. So the fragment of vm.args should be changed to:

## Name of the node
-name [email protected]

## Cookie for distributed Erlang
-setcookie 74b87337454200d4d33f80c4663dc5e5

app.config

The other configuration file is app.config. The important fragment of this configuration file is:

{vbt, [
    {tracker_listen_port, 2910 },
    {tracker_ssl_listen_port, undefined }
]}

This fragment specifies the port where Viblast Tracker listens for incoming Websocket connections. By default, Viblast Tracker listens for Websocket connections on port 2910 and for secure Websocket connections on port 2913.

Our recommendation is to use Viblast Tracker behind HAProxy. So the HAProxy will be configured to listen on port 2910 for Websocket connections and on port 2913 for secure Websocket connections and forward these connections to the Tracker. We will configure the Tracker to listen on port 2911.

{vbt, [
    {tracker_listen_port, 2911 },
    {tracker_ssl_listen_port, undefined }
]}

Viblast Tracker Start/Stop Scripts

The bin directory contains control scripts used to view the Tracker status and to interact with the running executable.

The vbt executable has the following sub-commands:

• start - The start sub-command is used to start the Viblast Tracker server. The configuration from etc/ will be read and the tracker node will be started.
• stop - The stop sub-command is used to stop an already running Viblast Tracker server.
• ping - The ping sub-command is used to check whether or not the viblast tracker is running on this machine.

Viblast Tracker Cluster Control

The vbt-admin executable is used to control the running cluster of several tracker nodes. The command has the following sub-commands:

• ringready - The ringready sub-command is used to check whether or not all tracker nodes are in a functional state and agree on the same server configuration.

• member-status - The member-status sub-command is used to list detailed information about each participating node in the cluster and its status.

• cluster - The cluster sub-command is used to plan and commit changes to the currently running viblast tracker cluster. It has the following options:

  • plan - The plan option will show the current staged changes that are still not commited;

  • clear - The clear option will clear the current cluster plan - any uncommitted changes will be dropped;

  • commit - The commit option will execute the current cluster change plan. After this command has completed successfully, all of the planned changes will be committed;

  • join - The join command will attempt to connect the current node with an already running cluster. The connecting node must be a single node and not part of a cluster. The join operation will be added to the current cluster change plan;

  • leave [<node>] - The leave command will attempt to disconnect the given node from the already running cluster. If the command is used without the extra argument for node selection, the currently running node will be disconnected from the cluster. The leave operation will be added to the current cluster change plan.

Viblast Tracker Service Script

The viblast tracker comes with a vbt init.d script file which is used to control the viblast tracker as a service.

Viblast Aggregator Deployment

Viblast aggregator is distributed as a generic Linux tar package vba.tar.gz. The current build supports Ubuntu Linux 14.04 LTS. Builds for other Linux distributions are available on request.

Viblast Aggregator Node

To install, untar the package into a directory (for example in /opt). The created directory structure is:

vba
├── bin
├── data
├── erts-6.4
├── etc
├── lib
├── log
├── priv
└── releases

Aggregator configuration files are stored in vba/etc directory. Scripts for starting/stopping viblast aggregator are found in vba/bin directory. By default, the logs are stored in vba/log directory and vba/data contains some runtime data about the Viblast aggregator cluster. The location of these directories could be configured.

Viblast Aggregator Configuration Files

Viblast aggregator is configured using the following files: vba/etc/ ├── vm.args └── app.config

vm.args

A fragment of vm.args with default values is shown below:

## Name of the node
-name [email protected]

## Cookie for distributed Erlang
-setcookie vbt

These parameters are used when connecting Tracker and aggregator nodes or when building tracker cluster. If you plan to connect this viblast aggregator node to other Tracker or Aggregator nodes, you have to change these parameters.

The first parameter is node name. If the private IP address of the server is 172.16.0.18, then a suitable node name would be [email protected]. The second parameter is setcookie. This parameter is a shared secret key. All nodes that should be connected to one another must have the same cookie parameter. A suitable value for this parameter is some secret hash. In our example, we will set this to 74b87337454200d4d33f80c4663dc5e5.

Since our aggregator node should be connected to the viblast tracker node [email protected], these two nodes should share the same cookie.

So the fragment of vm.args should be changed to: ## Name of the node -name [email protected]

## Cookie for distributed Erlang
-setcookie 74b87337454200d4d33f80c4663dc5e5

app.config

The other configuration file is app.config. The important parameters that should be reviewed are:

%% vba config
{vba, [
    {tracker_nodes,             ['[email protected]'] },
    {graphite_host,             "172.16.0.254"},
    {graphite_port,             2003 },
    {graphite_prefix,           "viblast"},
    {mysql_hostname,            "172.16.0.253"},
    {mysql_username,            "viblast"},
    {mysql_password,            "<viblast mysql password>"},
    {mysql_database,            "viblast"}
]}

First, Viblast aggregator should know from which Viblast tracker nodes it should collect and aggregate data:

• tracker_nodes - list of tracker nodes from which this aggregator collects data.

Viblast aggregator stores time-series data into Graphite, so you need to specify where Graphite is installed:

• graphite_host - the server (hostname or IP address) where the Graphite installation lives

• graphite_port - the port where Graphite Carbon daemon listens for metric updates

• graphite_prefix - a common prefix for metrics stored by Viblast; use this to adjust Graphite configuration and storage schema

Viblast aggregator stores some additional metrics metadata into MySQL database and needs to know which MySQL server and database to use:

• mysql_hostname - server (hostname or IP address) where MySQL server is installed
• mysql_database - database allocated for storing Viblast metrics metadata
• mysql_username - a MySQL user to use for accessing Viblast database
• mysql_password - password for authenticating the MySQL user

Viblast Aggregator Start/Stop Scripts

The bin directory contains control scripts used to view the aggregator status and to interact with the running executable.

The vba executable has the following sub-commands: * start - The start sub-command is used to start the viblast aggregator server. The configuration from etc/ will be read and the aggregator node will be started. * stop - The stop sub-command is used to stop an already running viblast aggregator server. * ping - The ping sub-command is used to check whether or not the viblast aggregator server is running on this machine.

Viblast Aggregator Cluster Control

The vba-admin executable is used to control the running cluster of several aggregator nodes. The command has the following sub-commands:

• ringready - The ringready sub-command is used to check whether or not all aggregator nodes are in a functional state and agree on the same server configuration.
• member-status - The member-status sub-command is used to list a detailed information about each participating node in the cluster and it’s status.

• cluster - The cluster sub-command is used to plan and commit changes to the currently running viblast aggregator cluster. It has the following options:

  • plan - The plan option will show the current staged changes that are still not commited;
  • clear - The clear option will clear the current cluster plan - any uncommitted changes will be dropped;
  • commit - The commit option will execute the current cluster change plan. After this command has completed successfully, all of the planned changes will be committed;
  • join - The join command will attempt to connect the current node with an already running cluster. The connecting node must be a single node and not part of a cluster. The join operation will be added to the current cluster change plan;
  • leave [<node>] - The leave command will attempt to disconnect the given node from the already running cluster. If the command is used without the extra argument for node selection, the the currently running node will be disconnected from the cluster. The leave operation will be added to the current cluster change plan.

Viblast Aggregator Service Script

The viblast Aggregator comes with a vba init.d script file which is used to control the viblast Aggregator as a service.

HAProxy Configuration

After HAProxy is successfully installed, its configuration is usually located at /etc/haproxy/haproxy.cfg. Example HAProxy configuration is shown below:

global
    maxconn 200000

frontend tracker_frontend
    timeout client 30m
    maxconn 200000

    bind :2910
    bind :2913 ssl crt /etc/ssl/private/ssl-cert-snakeoil.pem

    use_backend tracker_backend

backend tracker_backend
    balance leastconn

    server vbt0 172.16.0.17:2911 maxconn 50000

The first thing that needs to be done is to increase the maximum connections limit. This is done by adding the maxconn option to the global section.

A frontend needs to be configured for the tracker. The frontend must also contain options for increasing the maximum connection limit. All ports that the HAProxy will listen to must be listed with the bind option. If ssl will be used on any of the ports, the needed certificates must be provided as well. Lastly, the frontend must select a default backend.

The viblast tracker backend should include a balance strategy: the recommended one consists of choosing a tracker node based on the minimum number of open connections. The backend must also list all the viblast tracker nodes, the ports they are listening on and any number of extra parameters (e.g. how many connections can this node handle).