AWS S3 v2 Library Usage and Examples

Category	Library	Library Version	neqto: Bridge	SPRESENSE
Integrations	AWS_S3_V2	2020-07-20 *(latest)*	v00.00.30+	v01.00.00+
Integrations	AWS_S3_V2	2020-06-30	v00.00.30+	v01.00.00+

For information on how to import neqto.js libraries, please refer to this documentation.

Abstracts

Methods()/Properties	Summary
`new AWS_S3()`	Creates an AWS_S3 instance.

This library uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html

{AWS_S3} Instance

Methods()/Properties	Summary
`.createHttpPath()`	Create the HTTP path using the given filename and file path
`.createHttpHost()`	Create the HTTP Host string using this AWS_S3 instance's configuration.
`.createXAmzHeaders()`	Convert any AWS API styled `params` into `x-amz-*` headers. Ignores `bucket` and `key`.
`.createHttpHeader()`	Create the authentication headers for the given arguments according to the AWS Signature v4 documentation
`.put()`	Send a PUT request, outputting the result to the callback function.
`.get()`	Send a GET request, outputting the result to the passed callback.
`.abortRequest()`	Abort the currently ongoing request.

Details

`new AWS_S3(config)`

Configuration

When instantiating the AWS_S3 object, the following configurations are required to be set by the user:

Name	Type	Default	Summary
`region`	string	MANDATORY	Region to be used when accessing a bucket
`accessKeyId`	string	MANDATORY	Access key id for authorization from AWS IAM
`secretKey`	string	MANDATORY	Secret key for authorization from AWS IAM
`bucket`	string	MANDATORY	Name of the S3 Bucket to operate on
`ca`	string	MANDATORY	The root CA to communicate with AWS
`timeout`	number	90000	The timeout of the AWS_S3 Instance in ms ver. 2020-07-20+

The configuration must be a JavaScript object, as the following:

var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: "<string>"
};

Once the configuration is set, it can be passed to AWS_S3 during instance creation.

var s3 = new AWS_S3(config);

Setter Methods

After creating the AWS_S3 object, each of the aforementioned configurations may be changed by its corresponding setter, with a passed in value of String type. The values for accessKeyId and secretKey may be retrieved from AWS IAM.

Setter	Summary
`.setRegion(value: string)`	Set the region for this AWS_S3 instance.
`.setBucket(value: string)`	Set the bucket for this AWS_S3 instance.
`.setAccessKeyId(value: string)`	Set the Access Key ID for this AWS_S3 instance, used for authentication.
`.setSecretKey(value: string)`	Set the Secret Key for this AWS_S3 instance, used for authentication.
`.setRootCA(value: string)`	Set the Root CA for this AWS_S3 instance, used for connecting to AWS. The suggested value can be seen below.
`.setRequestTimeout(value: Number)`	Set the timeout for this AWS_S3 instance. ver. 2020-07-20+

The configuration setters can also be chained, if they were all successful. Each setter returns an Exception when given an invalid value, instead of an AWS_S3 instance for method chaining.

`.setConfig(config)`

The setConfig function can be used to set or change multiple config options at the same time, and takes in a typical JavaScript object of key:value pairs as an argument.

s3.setConfig({
    region: "value",
    accessKeyId: "value",
    secretKey: "value",
    bucket: "value",
    ca: "value"
});

Instance Methods

`.createHttpPath(filePath,fileName)`

Create the HTTP path using the given filename and file path

Name	Type	Default	Summary
`filePath`	string	MANDATORY	The path to the file on the bucket
`fileName`	string	MANDATORY	The file name
return	string	-	i.e, `/path/to/file.txt`

`.createHttpHost()`

Create the HTTP Host string using this AWS_S3 instance's configuration.

Name	Type	Default	Summary
return	string	-	i.e, `bucket.s3.amazonaws.com`

`.createXAmzHeaders(params)`

Create the x-amz-* headers from the passed params.

Name	Type	Default	Summary
`params`	Object	MANDATORY	The AWS API `params` object to convert into `x-amz-*` headers.
return	Object	-	`params` converted into `x-amz-*` headers if they exist as `params` in the API docs.

`.createHttpHeader(host,path,method,bodyToHash,additionalHeaders)`

Create the authentication headers for the given arguments according to the AWS Signature v4 documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

Name	Type	Default	Summary
`host`	string	MANDATORY	HTTP Host
`path`	string	MANDATORY	HTTP Path
`method`	string	MANDATORY	HTTP Method
`bodyToHash`	string	MANDATORY	HTTP Body; `null` or `falsy` to pass UNSIGNED_PAYLOAD instead.
`additionalHeaders`	Object	MANDATORY	an Object with additional `x-amz-*` headers for signing
return	Object	-	`{ Authorization: [auth], x-amz-content-sha256: [contenthash], x-amz-date: [timestamp]}`

`.put(path,userHeaders,body,callback)`

Send a PUT request, passing the result to the callback function. This function uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html

Name	Type	Default	Summary
`path`	string	MANDATORY	the path to the desired resource. host is always `[bucket].s3.amazonaws.com`.
`userHeaders`	Object	MANDATORY	An Object with a "Content-Length" header (mandatory) and any additional HTTP/2 or `x-amz-*` headers.
`body`	Function	MANDATORY	Callback to retrieve the contents of the object. Should return data in up to 4KiB [String / ArrayBuffer] chunks until exhausted, then return null.
`callback(err, data)`	function	MANDATORY	The user's callback function, to handle the response.

`.get(path,callback)`

Send a GET request, passing the result to the callback function. This function uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html

Name	Type	Default	Summary
`path`	string	MANDATORY	the path to the desired resource. host is always `[bucket].s3.amazonaws.com`.
`callback(err, data)`	function	MANDATORY	The user's callback function, to handle the response.

`.abortRequest()`

Aborts the currently ongoing request.

Name	Type	Default	Summary
return	undefined	-	-

Response Object

This library uses an object called a Response to handle AWS server responses. Responses are always in the following format:

{
    statusCode: <Number>,
    statusMessage: <string>,
    body: <ArrayBuffer> or <string>
}

More details on statusCode and statusMessage can be seen in the HTTPs documentation for the appropriate device (neqto: Bridge | Spresense). response is the result of .read() from that documentation.

Exceptions

In this library, Exceptions are returned in two ways: as return values from the setters, and passed as err into the callback function for the HTTP methods.

Exceptions raised outside of an HTTP request take the following form:

{
    name: <string>,
    message: <string>
}

Exceptions raised from an HTTP request take the following form:

{
    errCode: <Number>,
    message: <string>
}

Exceptions which detect invalid configurations, such as a missing Content-Length header, will have a message property; Exceptions which are raised from the HTTP/S layer will not have this property. Exceptions raised in this way have more information available regarding their errCode parameter in the neqto: Bridge and Spresense neqto.js documentation.

The following is an example of how to handle the various forms of Exceptions:

var invalidConfig = {
    badKey: "badValue"
};
var s3 = new AWS_S3(invalidConfig);
// s3 now contains an array of errors
if (Array.isArray(s3)) {
    //handle config errors
} else {
    //s3 is good in this block
    var callback = function (err, data) {
        if (err) {
            // handle errors
            print(err.errCode);
            if (err.message) {
                print(err.message);
            }
        } else {
            // handle data
        }
    }
    s3.get('/example.txt', callback);
}

Object Usage Example

The following is an example for the CA config option:

var awsS3Ca = '-----BEGIN CERTIFICATE-----\nMIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ\nRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD\nVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX\nDTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y\nZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy\nVHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr\nmD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr\nIZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK\nmpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu\nXmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy\ndc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye\njl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1\nBE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3\nDQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92\n9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx\njkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0\nEpn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz\nksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS\nR9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp\n-----END CERTIFICATE-----\n';

Example 1: Put an Object to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", err.errCode, err.message);
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Create a large buffer (representing data generated from sensors)
 * and send it to AWS S3 with a custom tag and ACL parameter.
 */

 var path = '/path/to/resource.txt';
 var dataSize = 12 * 1024;
 var params = {
     "Tagging": "example=tag",
     "ACL": "bucket-owner-full-control",
     "ContentLength": dataSize.toString()
 }

 var buffer = new ArrayBuffer(dataSize);
 var bufferView = new Uint8Array(buffer);
 for (var b = 0; b < bufferView.length; b++) {
     bufferView[b] = (b % 255); // fill it with arbitrary data
 }

 var putObject = function(path, params, body, callback) {
     var headers = s3.createXAmzHeaders(params);
     var MAX_CHUNK_SIZE = 4 * 1024;
     var bodyIndex = 0;
     var getNextChunk = function() {
         var ret;
         if (bodyIndex >= dataSize) {
             ret = null;
         } else {
             ret = body.slice(bodyIndex, bodyIndex + MAX_CHUNK_SIZE);
         }
         if (ret) {
             bodyIndex += ret.length || ret.byteLength;
         }
         return ret;
     }
     s3.put(path, headers, getNextChunk, callback);
 }

 putObject(path, params, buffer, callback);

Example 2: Put Data from Storage to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", err.errCode, err.message);
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Store data on the filesystem, then send that stored data.
 */

var path = "/path/to/storage.txt";
var dataSize = 16384;
var params = {
    "ContentLength": dataSize.toString(),
}

var store = storage.open();
var head = "HEADER:";

// Create a test data string and save it to memory
var s = "";
// Up to 4KiB of data, to be compatible with put()
while(s.length < 1024) {
    s += "testdata";
}
for (var i = 0; i < 24; i++) {
    store.writeFrame(head, s);
}

// If there was an error during memory save, recover it.
if (!store.readFrame(head, 0)) {
    var ret = store.recoverFrame(head);
}

var putStringFromStorage = function (strStore) {
    var headers = s3.createXAmzHeaders(params);
    var getData;
    if (strStore) {
        if (strStore.getReadableSize() < dataSize) {
            return null;
        }
        if (head) {
            // Get total amount of data in frames with this head
            var frames = strStore.searchAllFrame(head, 0);
            if (frames < dataSize) { // not enough stored data
                return null;
            }
            // Get total number of frames with this head
            frames = strStore.searchAllFrame(head, 1);
            var frameIndex = -1;
            var frameDataCollectedSize = 0;
            var getData = function() {
                if (frameIndex++ < frames && frameDataCollectedSize < dataSize) {
                    var ret = strStore.searchFrame(head, frameIndex);
                    frameDataCollectedSize += ret.length;
                    return ret;
                } else {
                    return null;
                }
            }
        }
    }
    s3.put(path, headers, getData, callback);
}
putStringFromStorage(store);

Example 3: Put Custom Data to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", err.errCode, err.message);
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Send data to S3 using a custom getData method.
 */

 var path = "/path/to/custom.txt";
 var dataSize = 256;
 var params = {
     "Content-Length": dataSize.toString()
 }

 var send = false;
 var getData = function () { // must return String, ArrayBuffer, or null.
     if (send) {
         return null;
     } else {
         send = true;
         var i = 0;
         var ret = "";
         while (i++ < dataSize) {
             ret += String.fromCharCode((i % 122) + 32); // ASCII table
         }
         return ret;
     }
 }

 s3.put(path, params, getData, callback);

Updated: 2020/07/20

Hardware Specifications

Software Specifications

neqto.js