# MediaDecoder : Multi Media decoder

The mediadecoder module provides multimedia decode and re-remux function.

The decode function can decode audio and video streams in multimedia files or streaming media, allowing user to obtain decoded audio and video frames.

The re-remux function can re-package multimedia files or streaming media according to the specified format, such as converting the RTSP protocol network camera to flv stream or mp4 format file.

User can use the following code to import the MediaDecoder module.

var MediaDecoder = require('mediadecoder');

# Support

The following shows MediaDecoder module APIs available for each permissions.

# MediaDecoder Class

# new MediaDecoder()

• Returns: {Object} mediadecoder object.

Create a new MediaDecoder object.

# MediaDecoder.open(url[, opts[, timeout[, callback]]])

Is equivalent to the following code:

MediaDecoder.open = function(url, opts, timeout, callback) {
  try {
    var mediadecoder = new MediaDecoder().open(url, opts, timeout, callback);
  } catch (error) {
    var mediadecoder = undefined;
  }
  return mediadecoder;
}

# MediaDecoder Object

# MediaDecoder Object status

The following table shows the possible status of mediadecoder object.

                         ┌─────────┐
      ┌──────────────────┤wait_open│
      │                  └─────────┘
      │                     Open
      │                       │
      │                       ▼
      │                  ┌─────────┐
      ├─────────Close────┤  Opened │
      │                  └─────────┘
      │                     Start
      ▼                       │
┌───────────┐                 ▼
│           │            ┌─────────┐
│   Closed  │◄──Close────┤ Started │ ◄──┐
│           │            └─────────┘    │
└───────────┘               Stop        │
      ▲                       │         │
      │                       ▼         │
      │                  ┌─────────┐    │
      ├─────Close────────┤ wait_eof│    │
      │                  └────┬────┘    │
      │                   Recv│eof      │
      │                       │         │
      │                       ▼         │
      │                  ┌─────────┐    │
      └─────Close────────┤ Stopped │    │
                         └────┬────┘    │
                              │         │
                              └──Start──┘

# mediadecoder.open(url[, opts[, timeout[, callback]]])

• url {String} The url of multimedia source.
• opts {Object} Open options.
• timeout {Integer} Open timeout time, the unit is milliseconds, default: 5000 milliseconds.
• callback {Function} Asynchronous callback function. default: undefined
  • error {Error} Open result, undefined mean open success, otherwise mean open failed.
• Returns: {MediaDecoder} Current object.

Open the multimedia source with given url, The MediaDecoder object can provide services for multitasking. You can use opt.name to specify the global name of this object, so that the subtask can open the same multimedia source by this name, so that the decoded video and audio can be shared, avoid decoding the same multimedia source to reduce the CPU load.

The MediaDecoder object can connect to the multimedia source in synchronous or asynchronous mode. If the callback function is specified, The open function will return immediately, and JSRE will notify the opening result through the specified callback. If no callback is specified, the open function will be blocked until the multimedia source have been opened successfully or the opening error occurs(such as timeout).

An exception will be generated when the open function fails to open multimedia source.

The options object contains the following members:

• name {String} The name of global media decoder object.
• proto {String} The transfer protocol of streaming media.

proto is a string, can be:

# Example

• Synchronous

var mediadecoder = new MediaDecoder();

mediadecoder.open('rtsp://user:passwd@ipaddr', {
  name: 'camera', proto: 'tcp'
  }, 5000);

• Asynchronous

var mediadecoder = new MediaDecoder();

mediadecoder.open('rtsp://user:passwd@ipaddr', {
  name: 'camera', proto: 'tcp'
  }, 5000, function(error) {
  if (error) {
    console.error('Open file', error);
  } else {
    console.info('Opened!');
  }
});

Subtasks can open the same multimedia source in the following ways:

# Example

var mediadecoder = new MediaDecoder();

mediadecoder.open('camera');

If we need to perform complex processing on multimedia, such as AI computing, video recording, screen capture, etc., we can do it through multi-tasking, so that the multi-core CPU has a higher degree of parallelization, and avoid decoding the same multimedia source.

# mediadecoder.close()

Close the media decoder, the mediadecoder object can't use again.

# mediadecoder.info()

• Returns: {Object} The information of multimedia source which has been opened.

Get the information of multimedia source which has been opened. The returns object contains the following members:

• duration {Integer} Media max durations, unit in milliseconds.
• video {Array} Video track information array.
• audio {Array} Audio track information array.
• subtitle {Array} Subtitle track information array.
• chapter {Array} Chapter information array.

The video track information object contains the following members:

• id {Integer} The id of video track.
• duration {Integer} Media durations, unit in milliseconds.
• codec {String} The codec name of video track, such as h264.
• lang {String} The language of video track.
• title {String} The title of video track.
• bitrate {Integer} The bitrate of video track, unit in Kbits/S.
• w {Integer} The width of video frame of video track.
• h {Integer} The height of video frame of video track.
• fps {Number} The fps of video frame of video track.

The audio track information object contains the following members:

• id {Integer} The id of audio track.
• duration {Integer} Media durations, unit in milliseconds.
• codec {String} The codec name of audio track, such as aac.
• lang {String} The language name of audio track.
• title {String} The title of audio track.
• bitrate {Integer} The bitrate of audio track, unit in Kbits/S.
• sampleRate {Integer} The sample rate of audio track, such as 44100.
• sampleBits {Integer} The sample bits of audio track, such as 16.
• channels {Integer} The channel number of audio track, such as 2.
• channelLayout {String} The channel layout of audio track, such as stereo.

channelLayout is a string, can be:

The subtitle track information object contains the following members:

• id {Integer} The id of subtitle track.
• duration {Integer} Media durations, unit in milliseconds.
• codec {String} The codec name of subtitle track, such as ass.
• lang {String} The language name of subtitle track.
• title {String} The title of subtitle track.
• bitrate {Integer} The bitrate of subtitle track, unit in Kbits/S.
• w {Integer} The width of canvas of subtitle track, only valid for graphic subtitles.
• h {Integer} The height of canvas of subtitle track. only valid for graphic subtitles.

The video, audio and subtitle track information object all have the following information:

• author {String} Author.
• album {String} Album.
• genre {String} Genre.
• composer {String} Composer.
• copyright {String} Copyright.
• comment {String} Comment.

The chapter information object contains the following members:

• id {Integer} The id of chapter.
• start {Integer} The start time of chapter, unit in milliseconds.
• end {Integer} The end time of chapter, unit in milliseconds.

duration is 0 which means media decoder cannot obtain, for example it might be a live stream or picture.

# Example

var MediaDecoder = require('mediadecoder');

var mediadecoder = new MediaDecoder();

var image = mediadecoder.open('file:///apps/test.jpg');
if (image == undefined) {
  console.error('Can not open image file!');
  return;
}

console.log(JSON.stringify(image.info()));

image.close();

var video = mediadecoder.open('file:///apps/test.mp4');
if (video == undefined) {
  console.error('Can not open video file!');
  return;
}

console.log(JSON.stringify(video.info()));

video.close();

mediadecoder = undefined;

# mediadecoder.on(event, listener)

• event {String} The name of the event.
• listener {Function} The callback function of the event.
• Returns: {Emitter} This emitter.

Adds the listener callback function to the end of the listener's list of mediadecoder object for the given event.

event is a string, can be:

# mediadecoder.start()

• Returns: {Boolean} Return true if success, otherwise false.

Start the media decoder. After call start successfully, The media decoder will start to read the multimedia source and execute decoding or re-muxing. The corresponding listener callback function will be called and pass the corresponding event data.

# mediadecoder.stop()

• Returns: undefined if success, generated an exception if failed.

Stop the media decoder, the status of media decoder will be changed to wait_eof after call stop successfully, On wait_eof status, the video and audio events will no longer be sent, but remux events will still be sent, until the eof event is sent, the status of media decoder will be changed to stopped.

# mediadecoder.srcVideoFormat()

• Returns: {Object} Source video format, return undefined if multimedia source without video stream.

Get the source video format of media decoder.

The returned object contains the following members:

• width {Integer} Video width.
• height {Integer} Video height.
• pixelFormat {Integer} Video pixel format.
• fps {Integer} Video frame rate.

pixelFormat is a integer, can be:

# mediadecoder.destVideoFormat(fmt)

• fmt {Object} Destination Video format.
• Returns: {Boolean} Return true if success, otherwise false.

Set the destination video format of media decoder.

The destination video format object contains the following members:

• disable {Boolean} Does disable video?
• width {Integer} Video width.
• height {Integer} Video height.
• pixelFormat {Integer} Video pixel format.
• fps {Integer} Video frame rate.

# mediadecoder.srcAudioFormat()

• Returns: {Object} Source audio format, return undefined if multimedia source without audio stream.

Get the source audio format of media decoder.

The returned object contains the following members:

• channelLayout {String} The layout of audio channel.
• channels {Integer} The number of audio channel.
• sampleRate {Integer} Audio sample rate.
• sampleFormat {Integer} Audio frame format.

sampleFormat is a integer, can be:

# mediadecoder.destAudioFormat(fmt)

• fmt {Object} Destination audio format.
• Returns: {Boolean} Return true if success, otherwise false.

Set the destination audio format of media decoder.

The destination audio format object contains the following members:

• disable {Boolean} Does disable audio?
• channelLayout {String} The layout of audio channel.
• channels {Integer} The number of audio channel.
• sampleRate {Integer} Audio sample rate.
• sampleFormat {Integer} Audio frame format.

# mediadecoder.previewFormat(fmt)

• fmt {Object} Destination preview video format.
• Returns: {Boolean} Return true if success, otherwise false.

Set the preview video format of media decoder.

The destination preview video format object contains the following members:

• disable {Boolean} Does disable preview?
• fb {Integer} The number of framebuffer device, 0 means channel 0.
• fps {Integer} Preview video frame rate.

# mediadecoder.previewBuffer()

• Returns: {Object} Preview video buffer.

Get the preview video buffer of media decoder.

The preview video buffer contains the following members:

• arrayBuffer {Buffer} Preview video buffer.
• width {Integer} Preview window width.
• height {Integer} Preview window height.
• pixelBytes {Integer} Bytes of preview video buffer pixel.
• pixelBits {Integer} Bits of preview video buffer pixel.
• rowBytes {Integer} Bytes of preview video buffer row.
• pixelFormat {Integer} Preview video buffer pixel format.

# mediadecoder.overlay()

• Returns: {Object} Preview video overlay object.

Get preview video overlay object. For more information, please refer to: overlay

The output of overlay will be superimposed on the preview output.

# mediadecoder.remuxFormat(fmt)

• fmt {Object} Remux format.
• Returns: {Boolean} Return true if success, otherwise false.

Set the remux format of media decoder. Remux data is mainly used for live streaming server and video recording.

The remux format object contains the following members:

• enable {Boolean} Does enable remux?
• enableAudio {Boolean} Does enable audio?
• audioFormat {String} Audio compression format.
• format {String} Destination media format, default: flv format.

format support flv format and mp4 format. mp4 format is available on EdgerOS 2.0.4 and above.

audioFormat specifies the audio compression format. Currently supports AAC('aac') or the same as the audio source. (e.g. G.711)

# MediaDecoder Events

# eof

• info {Object} Information.

The eof event will be sent in the following two cases:

• The application actively calls the stop function. When the media decoder enters the stop status from the wait_eof status, it means that other events (such as audio, video, remux) have ended.
• The multimedia source have been end or loss connection to the streaming media.

The info arguments is supported in EdgerOS 2.1.7 and above. The info object contains the following information:

• passive {Boolean} Whether to passively stopped.

# Example

image.on('eof', () => {
  quited = true;
});

image.start();

while (!quited) {
  iosched.poll(); // Event poll.
}

# header

The multimedia data stream that has been remuxed with video and audio, the typical format is H.264 + AAC / G.711 data stream, which can be used for video recording or live streaming service through WebMedia.

When enable remux and start the media decoder, the header and remux event will be generated. If video recording is required, this header data needs to be written in the header of the video recording file.

The remux header event object contains the following members:

• arrayBuffer {Buffer} The remux header data buffer.
• offset {Integer} The remux header data file offset.

# Example

var f = fs.open('./media.flv', 'c+');

image.on('header', (header) => {
  var buf = Buffer.from(header.arrayBuffer);
  f.write(buf, 0, buf.byteLength, pkt.offset);
});

# remux

The remux event object contains the following members:

• arrayBuffer {Buffer} The remux data buffer.
• offset {Integer} The remux data file offset.

# Example

image.on('remux', (remux) => {
  var buf = Buffer.from(remux.arrayBuffer);
  f.write(buf, 0, buf.byteLength, pkt.offset);
});

# video

The video decodes data frames, and can perform snapshots or AI calculations. If the single-task CPU burdens the task too much, AI calculations and snapshots can be carried out in multiple tasks to make full use of the advantages of multi-core parallelization.

The video event object contains the following members:

• arrayBuffer {Buffer} The video frame data buffer.

The format of the video frame is the format set by the destVideoFormat function.

# Example

image.on('video', (video) => {
  var image = {
    width: ..., // Determined by destVideoFormat
    height: ...,
    components: ...,
    buffer: new Buffer(video.arrayBuffer)
  };

  imagecodec.encode(image, './test.jpg', { quality: 80 });
  // Or:
  var jpgBuf = imagecodec.encode(image, 'jpg', { quality: 80 });
});

# audio

Similar to the video event, this event contains audio data, and the included audio data is in uncompressed PCM format, which can be used for speech recognition in future.

The audio event object contains the following members:

• arrayBuffer {Buffer} The audio frame data buffer.

The format of the audio frame is the format set by the destAudioFormat function.

# Example

var f = fs.open('./audio.pcm', 'c+');

netcam.on('audio', (audio) => {
  f.write(new Buffer(audio.arrayBuffer));
});

# Example

This example show how to decode a local image file and preview it on /dev/fb0.

var MediaDecoder = require('mediadecoder');
var iosched = require('iosched');

var image = new MediaDecoder().open('file://./1.bmp');
if (image == undefined) {
  console.error('Can not open image file!');
  return;
}

image.destVideoFormat({width: 240, height: 240, fps: 1, pixelFormat: MediaDecoder.PIX_FMT_RGB24, noDrop: false, disable: false});
image.destAudioFormat({disable: true});
image.previewFormat({enable: true, fb: 0, fps: 25});

var quited = false;

image.on('video', (video) => {
  console.log('get video frame');
  // do something
});

image.on('eof', () => {
  quited = true;
});

image.start();

while (!quited) {
  iosched.poll(); // Event poll.
}

image.close();

This example show how to connect a rtsp protocol netcam and preview video on /dev/fb0.

var MediaDecoder = require('mediadecoder');
var iosched = require('iosched');

var netcam = new MediaDecoder().open('rtsp://admin:admin@10.4.0.12', {proto: 'tcp'}, 10000);
if (netcam == undefined) {
  console.error('Can not connect camera!');
  return;
}

netcam.destVideoFormat({width: 640, height: 360, fps: 1, pixelFormat: MediaDecoder.PIX_FMT_RGB24, noDrop: false, disable: false});
netcam.destAudioFormat({disable: true});
netcam.previewFormat({enable: true, fb: 0, fps: 25});

var quited = false;

netcam.on('video', (video) => {
  // do something
});

netcam.on('eof', () => {
  quited = true;
});

netcam.start();

while (!quited) {
  iosched.poll(); // Event poll.
}

netcam.close();

This example show how to use master and slave mediadecoder object remux a rtsp protocol netcam to flv stream and mp4 format file use multitasking method.

master mediadecoder source code:

var MediaDecoder = require('mediadecoder');
var iosched = require('iosched');

// Open network camera through url, and named camera 
var netcam = new MediaDecoder().open('rtsp://admin:admin@10.4.0.12', {name: 'camera', proto: 'tcp'}, 10000);
if (netcam == undefined) {
  console.error('Can not connect camera!');
  return;
}

// Disable video
netcam.destVideoFormat({disable: true}); 
// Disable audio
netcam.destAudioFormat({disable: true});
// Enable remux, remux format is flv
netcam.remuxFormat({enable: true, enableAudio: true, format: 'flv'}); 

var quited = false;

netcam.on('header', (pkt) => {
  // do something
});

netcam.on('remux', (pkt) => {
  // do something
});

netcam.on('eof', () => {
  // do something
  // Exit io sched
  quited = true;
});

// Start master mediadecoder
netcam.start();

// Create mp4 recording task, pass name argument
var recordTask = new Task('./mp4_record.js', { name: 'camera' }, {
  directory: module.directory
});

while (!quited) {
  iosched.poll(); // Event poll.
}

// Close master mediadecoder
netcam.close();

slave mediadecoder source code(save the following code as file mp4_record.js):

var MediaDecoder = require('mediadecoder');
var iosched = require('iosched');
var fs = require('fs');

// Open network camera through argument name
var netcam = new MediaDecoder().open(ARGUMENT.name);
if (netcam == undefined) {
  console.error('Can not connect camera!');
  return;
}

// Disable video
netcam.destVideoFormat({disable: true});
// Disable audio
netcam.destAudioFormat({disable: true});
// Enable remux, remux format is mp4, audio format is aac
netcam.remuxFormat({enable: true, enableAudio: true, audioFormat: 'aac', format: 'mp4'});

var quited = false;

// Create mp4 file
var mp4File = fs.open(`./test.mp4`, 'c');

netcam.on('header', (pkt) => {
  console.log('start record!');
  // Write remux header into mp4 file
  var buf = Buffer.from(pkt.arrayBuffer);
  mp4File.write(buf, 0, buf.byteLength, pkt.offset);
});

netcam.on('remux', (pkt) => {
  // Write remux data into mp4 file
  var buf = Buffer.from(pkt.arrayBuffer);
  mp4File.write(buf, 0, buf.byteLength, pkt.offset);
});

netcam.on('eof', () => {
  // Slave mediadecoder stopped, close mp4 file
  mp4File.close();
  // Exit io sched
  quited = true;
});

// Start slave mediadecoder, will begin recording
netcam.start();

// Set 10 second timeout
setTimeout(() => {
  console.log('stop record!');
  // Stop slave mediadecoder, will recv eof event
  netcam.stop();
}, 10000);

while (!quited) {
  iosched.poll(); // Event poll.
}

console.log('record exit!');

// Close slave mediadecoder
netcam.close();