# WebMedia : Web Media server

WebMedia mode is a streaming media service framework. It has the following characteristics:

Streaming media protocol supports:

• Support http-flv / ws-flv live media.
• Can extend other media protocols by create user media source.

Support for dual transmission channels:

• Stream channel use to push media stream.
• Data/event channel can communicate in duplex. The server can push data like subtitles to client and the client/server can send control event to each other.
• Web media server can work in STREAM mode(only stream channel) or COMPOUND mode(both stream and data channel) Independently.

Transport protocol supports:

• Stream channel support xhr(http/https) and ws(ws/wss) protocol.
• Data channel support ws(ws/wss) protocol.

User can use the following code to import the WebMedia module.

var WebMedia = require('webmedia');

# Support

The following shows WebMedia module APIs available for each permissions.

# Media Object

# WebMedia.createServer(opts[, ser])

• opts {Object} Options:
  • mode {Integer} 1 - STREAM mode: Only support stream channel. 2 - COMPOUND mode: support both stream and data/event channel, default: 1.
  • path {String} Media source url path, default: '/'.
  • connectLimits {Integer} Connection limits, default: 3.
  • mediaSource {Object} Media source options.
    • source {String} Media source type. 'flv' | 'rtsp_netcam' | [user defined media source].
    • inOpts {Object} Media source defined input options.
    • outOpts {Object} Media source defined output options.
  • streamChannel {Object} Stream channel options.
    • protocol {String} Transport protocol : xhr(http, https) | ws(ws, wss).
    • server {HttpServer | WsServer} Use outside HttpServer or WsServer as xhr stream channel proxy server.
    • saddr {Socket saddr} If server option not defined, create inside stream channel proxy server.
    • path {String} Media stream path, default to opts.path.
    • tls {TLS option} TLS options.
  • dataChannel {Object} Data channel options.
    • protocol {String} Transport protocol : ws(ws, wss).
    • server {WsServer} Use outside WsServer as ws data channel proxy server.
    • saddr {Socket saddr} If server option not defined, create inside data channel proxy server.
    • path {String} Media data channel path, default to opts.path.
    • tls {TLS option} TLS options.
• ser {HttpServer | WebApp} Web server. If `ser defined, user should not call 'mediaServer.start()' to start media server.
• Returns: {Object} Media server object.

This method creates a web media server. If mode set to COMPOUND mode, dataChannel options need to support.

When creating stream channel or data channel, if the server option is specified, the saddr option is invalid, otherwise the web server will be created internally using saddr. If both server and saddr options are not set, you need to bind the request handling function yourself. For more information, please see server.streamPipe() and server.dataPipe().

The mediaSource sets the media source parameters, and mediaSource.source determines the media source to be created. Media module provides two built-in media sources that support the http-flv | ws- FLV streaming protocol. For more information, please refer to Media Source.

# Example

• Create STREAM mode server:

var opts = {
  mode: 1,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'xhr',
  },
};

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var server = WebMedia.createServer(opts, app);
app.get('/live.flv', server.streamPipe);

app.start();

• Create COMPOUND mode server:

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var wsSer = WsServer.createServer('/live.flv', app);

var opts = {
  mode: 2,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'xhr',
  },
  dataChannel: {
    protocol: 'ws',
    server: wsSer,
  },
};
var server = WebMedia.createServer(opts, app);
app.get('/live.flv', server.streamPipe);

app.start();

• Create media server independently:

var opts = {
  mode: 1,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'xhr',
    saddr: socket.sockaddr(socket.INADDR_ANY, 8000),
  }
};
var server = WebMedia.createServer(opts);

server.start();

# WebMedia.registerSource(name, classType)

• name {String} Media source name.
• classType {MediaSource class type} MediaSource interface instantiation.
• Returns: {Boolean} True - register source class success.

Register user media source. See Extend Media Source.

# MediaServer Object

# server.source

• {Object} MediaSource object, refer to MediaSource Object.

# server.cliMgr

• {Object} ClientMgr object, refer to ClientMgr Object.

# server.streamPipe

• {Function} Stream channel request handle.

If the information for the stream channel's proxy server is not provided when creating the media server, the user needs to set up the server.streamPipe to receive client requests.

# Example

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var server = WebMedia.createServer(opts, app);
app.get('/demo', server.streamPipe);

# server.dataPipe

• {Function} Data channel request handle.

If the information for the data channel's proxy server is not provided when creating the media server, the user needs to set up the server.dataPipe to receive client requests.

# Example

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var wsSer = WsServer.createServer('/demo', app);
var server = WebMedia.createServer(opts, app);
wsSer.on('connection', server.dataPipe);

# server.start()

Start media server.

If an external server is bound when creating media server, there is no need to actively call server.start () to start media server.

# server.stop()

Stop media server.

# server.resume()

Change all client states to 'resume'. The media server is resume state when it starts.

# server.pause()

Change all client states to 'pause'. The media source should not push stream to clients.

# server.pushStream(chunk)

• chunk {Buffer} Stream chunk.

Users can use this method to push stream externally to media source, which is generally used by source without their own production data.

# server.sendStream([client, ]chunk)

• client {MediaClient} Media client object.
• chunk {Buffer} Media stream buffer.

Send stream to clients. If client is given, only send stream to given client. This method is usually called by media source.

The stream is transmitted through the stream channel. The stream is sent only when the stream channel is open and the running state is processed. See client.isRunning().

# server.sendData([client, ]opts, ]chunk)

• client {MediaClient} Media client object.
• opts {Object} Options of data.
• chunk {Object | String | Array} Media data.

Send data to clients. If client is given, only send data to given client. This method is usually called by media source. The data is sent only when the data channel is open and the running state is processed. See client.isRunning().

This method sends some structured data, such as subtitle information, to the client through data channel. The data protocol see Data Channel Protocol.

# server.sendEvent([client, ]event, ...args)

• client {MediaClient} Media client object.
• event {String} Event name.
• args {Arguments} Parameters that can be serialized to json string.

Send event to clients. If client is given, only send event to given client. This method is usually called by media source. The event is sent when the data channel is open. See client.isOpen().

This method sends some control event such as 'pause', 'resume', to the client through data channel. The data protocol see Data Channel Protocol.

# MediaServer Events

The first parameter of all events is media server itself.

# start

• server {MediaServer} Media server self.

Emit event when server start.

# stop

• server {MediaServer} Media server self.

Emit event when server stop.

# open

• server {MediaServer} Media server self.
• client {MediaClient} Media client.

Emit event when client open.

# close

• server {MediaServer} Media server self.
• client {MediaClient} Media client.

Emit event when client close.

# end

• server {MediaServer} Media server self.

Emit event when media stream end.

# pause

• server {MediaServer} Media server self.

Emit event when server pause.

# resume

• server {MediaServer} Media server self.

Emit event when server resume.

# ClientMgr Object

# cliMgr.count

• {Integer} Count of connection clients, limit by connectLimits option, see WebMedia.createServer().

# cliMgr.lookup(id)

• id {String | MediaClient} Client id or client object.
• Returns: {MediaClient | Undefined}

Look up media client. If not find, return undefined.

# iter([filter, ]cb)

• filter {Function} The filter callback.
  • client {MediaClient} Media client object.
  • Returns: {Boolean} False - client will not handle.
• cb {Function} Iter cakkback.

This method iter clients. Each valid client will be passed to the cb for processing.

# Example

this.cliMgr.iter(function(cli) {
  if (cli.isRunning()) {
    cli.sendData(opts, chunk);
  }
});

# MediaClient Object

# client.isRunning()

• Returns {Boolean} Mesia client is running or not.

The client can push stream and data while it's running.

# client.isPending()

• Returns {Boolean} Mesia client is pending or not.

Client is running, stream channel network is blocked, and stream will be cached. For applications with high real-time requirements, push stream should be stopped when client is pending.

# client.isOpen()

• Returns {Boolean} Mesia client is opend or not.

# client.close()

Close client.

# client.resume()

Resume client while client is not running. The client will change to running state.

# client.pause()

Pause client while client is running. The client will change to not running state.

# client.sendStream(chunk[, force])

• chunk {Buffer} Stream buffer.
• force {Boolean} True - force to send stream, default: false.

Push stream to client when client is running. Force push data when force set to true and client is opened.

# Example

if (client.isRunning()) {
  client.sendStream(buf);
}

# client.sendData([opts, ]chunk)

• opts {Object} Data options.
• chunk {Object | String | Array} Send data.

Send data to client by data channel when client is running.

Data protocol see Data Channel Protocol.

# Example

client.sendData({type: 'data'}, [{id: 1, x0: 0, y0: 0, x1: 100, y1: 100}]);

# client.send(opts, data[, cb])

• opts {Object | Undefined} Event options.
• data {String | Array | Object} Event data.
• cb {Function} Remote callback. Argument:
  • client {MediaClient} Media client self.
  • opts {Object | Undefined} Options reply by remote client.
  • data {String | Array | Object} Data reply by remote client.

Send message event to client by data channel when client is opened.If the cb is valid and the remote client replies to the event, cb will be called.

Event protocol see Data Channel Protocol.

# Example

• Send message event:

client.send({type: 'mediaInfo'}, {width: 480, height: 320});

• Send message event with reply:

client.send({type: 'mediaInfo'}, {width: 480, height: 320}, (client, opts, data) => {
  // Do something.
});

• Receive message event:

client.on('message', (client, opts, data, cb) => {
  // Do something.
  if (cb) {
    cb(opts2, data2); // cb(opts, data)
  }
});

# emit(event, ...args[, cb])

• event {String} Event name.
• args {String | Array | Object} Event args.
• cb {Function} Remote callback. Argument:
  • client {MediaClient} Media client self.
  • args {Any} Arguments reply by remote client.

Send event event to client by data channel when client is opened.If the cb is valid and the remote client replies to the event, cb will be called.

Users cannot use reserved events. Hold event:

• 'error', 'open', 'close', 'pause', 'resume', 'message'.

Event protocol see Data Channel Protocol.

# Example

• Send event:

client.emit('addRoles', {id: 1, name: 'xming', age: '22'}, {id: 2, name: 'xli', age: '25'});

• Send event with reply:

client.emit('addRoles', {id: 1, name: 'xming', age: '22'}, {id: 2, name: 'xli', age: '25'}, (client, result) => {
  if (result) {
    console.log('Add roles success.');
  }
});

• Receive event:

client.on('addRoles', (client) => {
  var args = Array.prototype.slice.call(arguments);
  for (var i = 1; i < args.length; i++) {
    var role = args[i];
    // Add role.
  }
});

• Receive event with reply:

client.on('addRole', (client, role, cb) => {
  // Add role.
  cb('ok'); // cb(...args)
});

# MediaClient Events

The first parameter of all events is media client itself.

# pause

• client {MediaClient} Media client self.

Emit pause event when client pause.

# resume

• client {MediaClient} Media client self.

Emit resume event when client resume.

# message

• client {MediaClient} Media client self.
• opts {Object | Undefined} Options send by remote client.
• data {String | Array | Object} Data send by remote client.

Emit message event when receive message event from remote.

# MediaSource Object

# source.server

• {MediaServer} Media server. Refer to MediaServer Object.

# source.mode

• {Integer} Media server mode. 1 - STREAM mode; 2 - COMPOUND mode. Refer to WebMedia.createServer(opts[, ser]).

# source.getCliMgr()

• Returns: {ClientMgr} Client manage object, see ClientMgr Object.

# source.sendStream(chunk)

• chunk {Buffer} Send stream buffer.

Send stream directly to all the currently connected clients by stream channel.

# source.sendData(opts, chunk)

• opts {Object} Send data options.
• chunk {String | Object | Array} Send data.

Send data directly to all the currently connected clients by data channel. The method only valid in COMPOUND mode.

# source.sendStreamHeader(chunk)

• chunk {Buffer} Send stream header.

Send the header information to all clients by stream channel. The header information will be cached, and the client will receive the header information whenever the client connects.

# source.sendDataHeader(opts, chunk)

• opts {Object} Send header data options.
• chunk {String | Object | Array} Send heaer data.

Send the header information to all clients by data channel. The header information will be cached, and the client will receive the header information whenever the client connects. The method only valid in COMPOUND mode.

# source.end()

End media source stream. This method wiil notify media server source end buf not stop source.

# source.start()

This method is an interface, and the user needs to implement the startup media source behavior.

# source.stop()

This method is an interface, and the user needs to implement the stop media source behavior.

# source.pushStream(chunk)

• chunk {Buffer} Push meida stream.

This method is an interface, and User could implementation this interface to receive stream data. If use push stream outside meida source by call MediaServer.pushStream(), user must implementation this interface.

# Media Source Events

# start

Emit event when source start.

In the implementation of MediaSource, this event must be emit when the source is started.

# stop

Emit event when source stop.

In the implementation of MediaSource, this event must be emit when the source is stopped.

# Media Channel

The media stream is pushed to the client through the stream channel, and the transmission of the stream data follows the corresponding media protocol. The rest of the user's structured data is transmitted through data channel, data channel defines the message container format, where users can customize parameters.

Users cannot use message events defined within the 'MEDIA' module. These events are:

• error
• open
• close
• pause
• resume
• data
• message

MediaClient object provides a set of methods for sending and receiving message, there are:

• client.sendData() Send data event.
• client.send() Send message event.
• client.emit() Send user event except build-in.

User can receive message by useclient.on(event, client, ...args) to handle callback.Special, the argument of message event as client.on(event, client, opts, data).

# Data Channel Protocol

The data is described in json sring format, and each message is a json object whose fields are defined as follows:

# Stream Mode Connection Process

           Client                                       Server
             |                                            |
             | [stream channel connect]                   |
             | [protocol]://[host]:[port]/[path]          |
             +------------------------------------------> +
             |                                            |
             | [stream channel push media stream]         |
             + <------------------------------------------+
             |                                            |

# Compound Mode Connection Process

           Client                                      Server
             |                                           |
             | [data channel connect]                    |
             | [protocol]://[host]:[port]/[path]         |
             +-----------------------------------------> +
             |                                           |
             | [data channel event]                      |
             | id: [ID]                                  |
             | type: 1                                   |
             | event: 'shakeHandle'                      |
             + <-----------------------------------------+
             |                                           |
             | [stream channel connect]                  |
             | [protocol]://[host]:[port]/[path]?id=[ID] |
             +-----------------------------------------> +
             |                                           |
             | [data channel event]                      |
             | id: [ID]                                  |
             | type: 1                                   |
             | event: 'open'                             |
             + <-----------------------------------------+
                               ......
             |                                           |
             | [stream channel push media stream]        |
             + <-----------------------------------------+
             |                                           |
             | [data channel event]                      |
             | id: [ID]                                  |
             | type: 1                                   |
             | event: 'data'                             |
             | opts: OPTS                                |
             | data: DATA                                |
             + <---------- client.sendData() ------------+
             |                                           |

# Call-Reply Process

• message event:

       Client/Server                                                        Server/Client
             |                                                                    |
             | [data channel event]                                               |
             | id: [ID]                                                           |
             | type: 2                                                            |
             | event: 'message'                                                   |
             | eventId: [EVENT_ID]                                                |
             | opts: OPTS1                                                        |
             | data: DATA1                                                        |
             + client.send(,cb) ---------> client.on('message',,function(,reply)) +
             |                                                                    |
             | [data channel event]                                               |
             | id: [ID]                                                           |
             | type: 3                                                            |
             | event: 'message'                                                   |
             | eventId: [EVENT_ID]                                                |
             | opts: OPTS2                                                        |
             | data: DATA2                                                        |
             + cb() <---------------------------------------------------- reply() +
             |                                                                    |

• User event:

       Client/Server                                                        Server/Client
             |                                                                    |
             | [data channel event]                                               |
             | id: [ID]                                                           |
             | type: 2                                                            |
             | event: [EVENT]                                                     |
             | eventId: [EVENT_ID]                                                |
             | data: [arg1, arg2,...,argn]                                        |
             + client.emit(EVENT,cb) --------> client.on(EVENT,,function(,reply)) +
             |                                                                    |
             | [data channel event]                                               |
             | id: [ID]                                                           |
             | type: 3                                                            |
             | event: [EVENT]                                                     |
             | eventId: [EVENT_ID]                                                |
             | data: [arg'1, arg'2,...,arg'n]                                     |
             + cb() <---------------------------------------------------- reply() +
             |                                                                    |

# Media Source

Currently, the built-in media source supports http-flv/ws-flv live streaming protocol, and users can also define their own media Source to support other streaming protocols.

# Built-In Media Source

The flv built-in media source supports http-flv/ws-flv live streaming protocol.

Create media server by setting mediaSource option to use flv built-in media source, more information to see media.createServer(opts[, ser]). flv built-in media source options:

• mediaSource {Object} Media source options.
  • source {String} 'flv'.
  • inOpts {Object} Media source defined input options.
    • netcam {Mediadecoder | Undefined} Media source will receive flv stream from netcam.repacket event. If netcam option not support, user should push flv stream by server.pushStream().

# Example

Create 'flv' source media server:

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));

var opts = {
  mode: 1,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'xhr',
  },
}
var server = WebMedia.createServer(opts, app);
app.get('/live.flv', server.streamPipe);

netcam.on('remux', (frame) => {
  var buf = Buffer.from(frame.arrayBuffer);
  server.pushStream(buf);
});

netcam.on('header', (frame) => {
  var buf = Buffer.from(frame.arrayBuffer);
  server.pushStream(buf);
});

app.start();
netcam.start();

# Extend Media Source

To extend the user's media source, you need to inherit the class MediaSource and implement its interface, and then register the new media source type into media server.

# Example

• Create new media source: demo_source.js

var MediaSource = require('webmedia').MediaSource;

class DemoSource extends MediaSource {
  constructor(ser, mode, inOpts, outOpts) {
    super(ser, mode, inOpts, outOpts);
    this._timer = undefined;
  }

  start() {
    var count = 1;
    var self = this;
    if (!this._timer) {
      this._timer = setInterval(function() {
        count++;
        var stream = Buffer.from(`This is test stream, count=${count}`);
        self.sendStream(stream);
        if (self.mode === 2) {
          self.sendData({type: 'data'}, `This is test data, count=${count}`);
        }
      }, 2000);
    }
    super.start.call(this);
  }

  stop() {
    if (this._timer) {
      clearInterval(this._timer);
      this._timer = undefined;
    }
    super.stop.call(this);
  }
}

• Use extend media source:

var WebMedia = require('webmedia');
var DemoSource = require('./demo_source');
var socket = require('socket');
var WebApp = require('webapp');
var WsServer = require('websocket').WsServer;

WebMedia.registerSource('demo', DemoSource);

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var wsSer = WsServer.createServer('/demo', app);

var opts = {
  mode: 2,
  path: '/demo',
  mediaSource: {
    source: 'demo',
  },
  streamChannel: {
    protocol: 'xhr',
  },
  dataChannel: {
    protocol: 'ws',
    server: wsSer,
  },
}
var server = WebMedia.createServer(opts, app);
app.get('/demo', server.streamPipe);

app.start();

# Example

# HTTP-FLV

• Server example:

User flv media source to push stream.

var socket = require('socket');
var iosched = require('iosched');
var WebApp = require('webapp');
var MediaDecoder = require('mediadecoder');
var WebMedia = require('webmedia');

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));

var opts = {
  mode: 1,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'xhr',
  },
}
var server = WebMedia.createServer(opts, app);

server.on('start', () => {
  var netcam = new MediaDecoder().open('rtsp://admin:admin@10.4.0.12', {proto: 'tcp'}, 10000);

  netcam.destVideoFormat({width: 640, height: 360, fps: 1, pixelFormat: MediaDecoder.PIX_FMT_RGB24, noDrop: false, disable: false});
  netcam.destAudioFormat({disable: false});
  netcam.remuxFormat({enable: true, enableAudio: true, format: 'flv'});
  netcam.on('remux', (frame) => {
    var buf = Buffer.from(frame.arrayBuffer);
    server.pushStream(buf);
  });
  netcam.on('header', (frame) => {
    var buf = Buffer.from(frame.arrayBuffer);
    server.pushStream(buf);
  });
  netcam.start();
});

app.get('/', function(req, res) {
  res.sendFile('./flv_http.html');
});
app.get('/live.flv', server.streamPipe);

app.start();

iosched.forever();

• Client player example: 'flv_http.html'

This example use flv.js player to play http-flv stream. For more information, please refer to flv.js.

<html>
<head>
  <link rel="icon" href="data:image/ico;base64,aWNv">
</head>
<body>
  <script src="https://cdn.bootcss.com/flv.js/1.5.0/flv.min.js"></script>
  <video id="videoElement"></video>
  <script>
    if (flvjs.isSupported()) {
      var videoElement = document.getElementById('videoElement');
      var flvPlayer = flvjs.createPlayer({
        enableStashBuffer: false,
        type: 'flv',
        isLive: true,
        url: `http://${document.domain}:${window.location.port}/live.flv`,
      });
      flvPlayer.attachMediaElement(videoElement);
      flvPlayer.load();
      flvPlayer.play();
    }
  </script>
</body>
</html>

# WS-FLV

• Server example:

User flv media source to push stream.

var socket = require('socket');
var iosched = require('iosched');
var WebApp = require('webapp');
var WsServer = require('websocket').WsServer;
var MediaDecoder = require('mediadecoder');
var WebMedia = require('webmedia');

var app = WebApp.create('media', 0, socket.sockaddr(socket.INADDR_ANY, 8000));
var wsSer = WsServer.createServer('/live.flv', app);

var opts = {
  mode: 1,
  path: '/live.flv',
  mediaSource: {
    source: 'flv',
  },
  streamChannel: {
    protocol: 'ws',
    server: wsSer
  },
}
var server = WebMedia.createServer(opts, app);

server.on('start', () => {
  var netcam = new MediaDecoder().open('rtsp://admin:admin@10.4.0.12', {proto: 'tcp'}, 10000);

  netcam.destVideoFormat({width: 640, height: 360, fps: 1, pixelFormat: MediaDecoder.PIX_FMT_RGB24, noDrop: false, disable: false});
  netcam.destAudioFormat({disable: false});
  netcam.remuxFormat({enable: true, enableAudio: true, format: 'flv'});
  netcam.on('remux', (frame) => {
    var buf = Buffer.from(frame.arrayBuffer);
    server.pushStream(buf);
  });
  netcam.on('header', (frame) => {
    var buf = Buffer.from(frame.arrayBuffer);
    server.pushStream(buf);
  });
  netcam.start();
});

app.get('/', function(req, res) {
  res.sendFile('./flv_ws.html');
});
app.get('/live.flv', server.streamPipe);

app.start();

iosched.forever();

• Client player example: 'flv_ws.html'

This example use flv.js player to play ws-flv stream. For more information, please refer to flv.js.

<html>
<head>
  <link rel="icon" href="data:image/ico;base64,aWNv">
</head>
<body>
  <script src="https://cdn.bootcss.com/flv.js/1.5.0/flv.min.js"></script>
  <video id="videoElement"></video>
  <script>
    if (flvjs.isSupported()) {
      var videoElement = document.getElementById('videoElement');
      var flvPlayer = flvjs.createPlayer({
        enableStashBuffer: false,
        type: 'flv',
        isLive: true,
        url: `ws://${document.domain}:${window.location.port}/live.flv`,
      });
      flvPlayer.attachMediaElement(videoElement);
      flvPlayer.load();
      flvPlayer.play();
    }
  </script>
</body>
</html>