# WebGet : Download files using the http protocol

The WebGet module use to get http data. It supports multipart request data, breakpoint continuation and other functions.

User can use the following code to import the WebGet module.

var WebGet = require('webget');

# Support

The following shows WebGet module APIs available for each permissions.

# WebGet Class

WebGet currently does not support downloading HTTP header option 'Transfer-Encoding' as 'chunked' type data.

# WebGet.size(url, callback[, options[, tlsOpt]])

• url {String} Http url.
• callback {Function} Web get size handler.
  • size {Integer | Error} The data size or Error object when get size fail.
• options {Object} Has the following properties:
  • domain {socket.AF_INET | socket.AF_INET6} If the url host is provided as a domain name, the domain name is resolved to an ipv4 or ipv6 address based on the domain.default: socket.AF_INET.
  • saddr {Object} Server socket address. default: Use url parameter resolution, if you request the same domain name multiple times, it is recommended to set this parameter after manual domain name resolution to speed up the request.
  • path {String} The get uri path, default: url parsed path.
  • timeout {Integer} The get timeout. If the get times out, a Error object will return by callback.
  • headers {Object} Http headers.
  • host {String} The domain name or IP address of the server to which the request is sent. default: url host.
• tlsOpt {Object} TLS securely connections options. default: undefined, means use TCP.

Get url data size.

# Example

var WebGet = require('webget');
var iosched = require('iosched');

var url = 'http://www.sylixos.com/img/logo.png';
WebGet.size(url, (size) => {
  if (size instanceof Error) {
    console.error(size);
  } else {
    console.log(`Get res(${url}) size:`, size);
  }
});

while(true) {
  iosched.poll();
}

# WebGet.get(url[, options], callback[, tlsOpt])

• url {String} Http url.
• options {Object} Has the following properties:
  • domain {socket.AF_INET | socket.AF_INET6} If the url host is provided as a domain name, the domain name is resolved to an ipv4 or ipv6 address based on the domain.default: socket.AF_INET.
  • saddr {Object} Server socket address. default: Use url parameter resolution, if you request the same domain name multiple times, it is recommended to set this parameter after manual domain name resolution to speed up the request.
  • path {String} The get uri path, default: url parsed path.
  • timeout {Integer} The get timeout. If the get times out, the webget object will emit a error event.
  • headers {Object} Http headers.
  • host {String} The domain name or IP address of the server to which the request is sent. default: url host.
  • begin {Integer} The begin position of load data. default: 0.
  • end {Integer} The end position of load data, 0 - end of data. default: 0.
  • limits {Integer} Data size limits per get range. default: 5MByts.
  • lines {Integer} The max count of requests at same time. default: 1 max: 4.
• callback {Function} Web get handler.
  • loader {WebGet} The WebGet object.
• tlsOpt {Object} TLS securely connections options. default: undefined, means use TCP.

Http get data. The user can specify the starting position of the data, the size of each segment of data, and the number of parallel requests in the options. The callback will retun WebGet object. Users can get data and other information in the events of the WebGet object.

# Example

var WebGet = require('webget');
var iosched = require('iosched');
var fs = require('fs');

var path = `./logo.png`;
var file = fs.open(path, 'w');
if (!file) {
  throw new Error('Open file error, file:', path);
}

WebGet.get('http://www.sylixos.com/img/logo.png', {
  limits: 1024,
  lines: 2,
}, (loader) => {
  loader.on('response', (info) => {
    console.log(`Download begin, original=${info.originalSize}, total=${info.requestSize}.`);
  });

  loader.on('data', (chunk, info) => {
    console.log(`Recv data, size=${chunk.byteLength}, offset=${info.offset}.`);
    file.write(chunk, 0, chunk.byteLength, info.offset);
  });

  loader.on('end', () => {
    console.log(`Download finish, file: ${path}`);
    file.close();
  });

  loader.on('error', (e) => {
    console.log('Download error:', e);
    file.close();
  });
});

while(true) {
  iosched.poll();
}

# WebGet.file(url, filePath[, options], callback[, tlsOpt])

• url {String} Http url.
• filePath {String} The file path to which the data is saved, including the file name. If the file exists and reload option is setted, 'WebGet' object will check the log and start the breakpoint resume process.
• options{Object} Has the following properties:
  • domain {socket.AF_INET | socket.AF_INET6} If the url host is provided as a domain name, the domain name is resolved to an ipv4 or ipv6 address based on the domain.default: socket.AF_INET.
  • saddr {Object} Server socket address. default: Use url parameter resolution, if you request the same domain name multiple times, it is recommended to set this parameter after manual domain name resolution to speed up the request.
  • path {String} The get uri path, default: url parsed path.
  • timeout {Integer} The get timeout. If the get times out, the webget object will emit a error event.
  • headers {Object} Http headers.
  • host {String} The domain name or IP address of the server to which the request is sent. default: url host.
  • begin {Integer} The begin position of load data. default: 0.
  • end {Integer} The end position of load data, 0 - end of data. default: 0.
  • limits {Integer} Data size limits per get range. default: 5MByts.
  • lines {Integer} The max count of requests at same time. default: 1 max: 4.
  • reload {Boolean} true: Delete the downloaded file and download it again. false: If the file exists, 'WebGet' object will check the log and start the breakpoint resume process. default: false.
• callback {Function} Web get handler.
  • loader {WebGet} The WebGet object.
• tlsOpt {Object} TLS securely connections options. default: undefined, means use TCP.

Http get data and save it to file. The user can specify the starting position of the data, the size of each segment of data, and the number of parallel requests in the options. If the file exists and reload option is setted, 'WebGet' object will check the log and start the breakpoint resume process. The callback will retun WebGet object. Users can get data and other information in the events of the WebGet object.

# Example

var WebGet = require('webget');
var iosched = require('iosched');

var url = 'http://www.sylixos.com/img/logo.png';
var path = './logo.png';
var totalSize = 0;

WebGet.file(url, path, {
  limits: 512,
  lines: 2,
  reload: true,
}, (loader) => {
  loader.on('response', (info) => {
    totalSize = info.requestSize;
    console.log(`Download begin, original=${info.originalSize}, total=${totalSize}, loaded=${info.loadedSize}`);
  });

  loader.on('data', (chunk, info) => {
    var progress = info.completeSize / totalSize * 100;
    console.log(`Recv data, size=${chunk.byteLength}, offset=${info.offset}, progress=${progress}%`);
  });

  loader.on('end', () => {
    console.log(`Download finish, file: ${path}`);
  });

  loader.on('error', (e) => {
    console.log('Download error:', e);
  });
});

while(true) {
  iosched.poll();
}

# WebGet Object Event

# response

A response event is emitted when the webget response is coming. It has a info properties:

• info {Object} It has options:
  • originalSize {Integer} The original url data size.
  • requestSize {Integer} The reuqest's(begin and end options defined) data size.
  • loadedSize {Integer} The size of the data that has been obtained.

# data

A data event is emitted with a data. It has follow properties:

• chunk {Buffer} Receive data.
• info {Object} It has options:
  • offset {Integer} The offset of the data in the requested data.
  • completeSize {Integer} The size of the data that has been received.

# end

A end event is emitted when the data get done.

# error

A close event is emitted when an error occurs. The WebGet object will destory after error occurs.