AWS S3 v2 Library

Category	Library	Library Version	neqto: Bridge	SPRESENSE
Integrations	AWS_S3_V2	2020-07-20 (latest)	v00.00.30+	v01.00.00+
Integrations	AWS_S3_V2	2020-06-30	v00.00.30+	v01.00.00+

For information on how to import neqto.js libraries, please refer to this documentation.

Abstracts

Methods()/Properties	Summary
new AWS_S3()	Creates an AWS_S3 instance.

This library uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html

{AWS_S3} Instance

Methods()/Properties	Summary
.createHttpPath()	Create the HTTP path using the given filename and file path
.createHttpHost()	Create the HTTP host string using this AWS_S3 instance's configuration.
.createXAmzHeaders()	Convert any AWS API styled params into x-amz-* headers. Ignores bucket and key.
.createHttpHeader()	Create the authentication headers for the given arguments according to the AWS Signature v4 documentation
.put()	Send a PUT request, outputting the result to the callback function.
.get()	Send a GET request, outputting the result to the passed callback.
.abortRequest()	Abort the currently ongoing request.

Details

new AWS_S3(config)

Initializes the AWS_S3 object with the passed configuration.

Configuration

When instantiating the AWS_S3 object, the following configurations are required to be set by the user:

Name	Type	Default	Summary
region	string	MANDATORY	Region to be used when accessing a bucket
accessKeyId	string	MANDATORY	Access key id for authorization from AWS IAM
secretKey	string	MANDATORY	Secret key for authorization from AWS IAM
bucket	string	MANDATORY	Name of the S3 Bucket to operate on
ca	string	MANDATORY	The root CA to communicate with AWS
timeout	number	90000	The timeout of the AWS_S3 Instance in ms ver. 2020-07-20+

The configuration must be a JavaScript object, as the following:

var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: "<string>"
};

Once the configuration is set, it can be passed to AWS_S3 during instance creation.

var s3 = new AWS_S3(config);

Setter Methods

After creating the AWS_S3 object, each of the aforementioned configurations may be changed by its corresponding setter, with a passed in value of String type. The values for accessKeyId and secretKey may be retrieved from AWS IAM.

Setter	Summary
.setRegion(value: string)	Set the region for this AWS_S3 instance.
.setBucket(value: string)	Set the bucket for this AWS_S3 instance.
.setAccessKeyId(value: string)	Set the Access Key ID for this AWS_S3 instance, used for authentication.
.setSecretKey(value: string)	Set the Secret Key for this AWS_S3 instance, used for authentication.
.setRootCA(value: string)	Set the root CA for this AWS_S3 instance, used for connecting to AWS. The suggested value can be seen below.
.setRequestTimeout(value: Number)	Set the timeout for this AWS_S3 instance. ver. 2020-07-20+	`

The configuration setters can also be chained, if they were all successful. Each setter returns an Error when given an invalid value, instead of an AWS_S3 instance for method chaining.

.setConfig(config)

The setConfig function can be used to set or change multiple config options at the same time, and takes in a typical JavaScript object of key:value pairs as an argument.

s3.setConfig({
    region: "value",
    accessKeyId: "value",
    secretKey: "value",
    bucket: "value",
    ca: "value"
});

Instance Methods

.createHttpPath(filePath,fileName)

Create the HTTP path using the given filename and file path

Name	Type	Default	Summary
filePath	string	MANDATORY	The path to the file on the bucket
fileName	string	MANDATORY	The file name
return	string	-	I.e. /path/to/file.txt

.createHttpHost()

Create the HTTP host string using this AWS_S3 instance's configuration.

Name	Type	Default	Summary
return	string	-	I.e. bucket.s3.amazonaws.com

.createXAmzHeaders(params)

Create the x-amz-* headers from the passed params.

Name	Type	Default	Summary
params	Object	MANDATORY	The AWS API params object to convert into x-amz-* headers.
return	Object	-	params converted into x-amz-* headers if they exist as params in the API docs.

.createHttpHeader(host,path,method,bodyToHash,additionalHeaders)

Create the authentication headers for the given arguments according to the AWS Signature v4 documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

Name	Type	Default	Summary
host	string	MANDATORY	HTTP host
path	string	MANDATORY	HTTP path
method	string	MANDATORY	HTTP Method
bodyToHash	string	MANDATORY	HTTP Body; null or falsy to pass UNSIGNED_PAYLOAD instead.
additionalHeaders	Object	MANDATORY	An Object with additional x-amz-* headers for signing
return	Object	-	{ Authorization: [auth], x-amz-content-sha256: [contenthash], x-amz-date: [timestamp]}

.put(path,userHeaders,body,callback)

Send a PUT request, passing the result to the callback function. This function uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html

Name	Type	Default	Summary
path	string	MANDATORY	The path to the desired resource. The path is always appended to the host [bucket].s3.amazonaws.com.
userHeaders	Object	MANDATORY	An Object with a "Content-Length" header (mandatory) and any additional HTTP/2 or x-amz-* headers.
body	Function	MANDATORY	Callback to retrieve the contents of the object. Should return data in up to 4KB[String / ArrayBuffer] chunks until exhausted, then return null.
callback(err, resp)	function	MANDATORY	The user’s callback function, to handle the response. Will pass an Error as err and a Response as resp.

.get(path,callback)

Send a GET request, passing the result to the callback function. This function uses the AWS S3 API: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html

Name	Type	Default	Summary
path	string	MANDATORY	The path to the desired resource. The path is always appended to the host [bucket].s3.amazonaws.com.
callback(err, resp)	function	MANDATORY	The user’s callback function, to handle the response. Will pass an Error as err and a Response as resp.

.abortRequest()

Aborts the currently ongoing request.

Name	Type	Default	Summary
return	undefined	-	-

Response Object

This library uses an object called a Response to handle AWS server responses. Responses are always in the following format:

{
    statusCode: <Number>,
    statusMessage: <string>,
    body: <ArrayBuffer> or <string>
}

More details on statusCode and statusMessage can be seen in the HTTPs documentation for the appropriate device (neqto: Bridge | Spresense).

body is the body data from the HTTP response.

Errors

In this library, Errors are returned in two ways: as return values from the setters, and passed as err into the callback function for the HTTP methods.

Errors raised outside of an HTTP request take the following form:

{
    name: <string>,
    message: <string>
}

Errors raised from an HTTP request take the following form:

{
    errCode: <Number>
}

Errors which detect invalid configurations, such as a missing Content-Length header, will have a message property; Errors which are raised from the HTTP/S layer will not have this property. Errors raised in this way have more information available regarding their errCode parameter in the neqto: Bridge and Spresense neqto.js documentation.

The following is an example of how to handle the various forms of Errors:

var invalidConfig = {
    badKey: "badValue"
};
var s3 = new AWS_S3(invalidConfig);
// s3 now contains an array of errors
if (Array.isArray(s3)) {
    //handle config errors
} else {
    //s3 is good in this block
    var callback = function (err, data) {
        if (err) {
            // handle errors
            JSON.stringify(err)
        } else {
            // handle data
        }
    }
    s3.get('/example.txt', callback);
}

Object Usage Examples

The following is an example CA which can be used in the config option (as of 2020-07-20):

/* Baltimore CyberTrust Root - 020000b9 */
var awsS3Ca = '-----BEGIN CERTIFICATE-----\nMIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ\nRTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD\nVQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX\nDTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y\nZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy\nVHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr\nmD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr\nIZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK\nmpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu\nXmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy\ndc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye\njl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1\nBE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3\nDQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92\n9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx\njkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0\nEpn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz\nksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS\nR9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp\n-----END CERTIFICATE-----\n';

Verify that the root CA is appropriate for the target S3 endpoint.

Please refer to the following resources:

• Baltimore CyberTrust Root
• AWS Official Announcement
• Amazon Trust Services
  • Migration Schedule
  • How to Prepare

Example 1: Put an Object to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", JSON.stringify(err));
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Create a large buffer (representing data generated from sensors)
 * and send it to AWS S3 with a custom tag and ACL parameter.
 */

 var path = '/path/to/resource.txt';
 var dataSize = 12 * 1024;
 var params = {
     "Tagging": "example=tag", // Requires the s3:PutObjectTagging permission
     "ACL": "bucket-owner-full-control", // Requires the s3:PutObjectAcl permission
     "ContentLength": dataSize.toString()
 }

 var buffer = new ArrayBuffer(dataSize);
 var bufferView = new Uint8Array(buffer);
 for (var b = 0; b < bufferView.length; b++) {
     bufferView[b] = (b % 255); // Fill it with arbitrary data
 }

 var putObject = function(path, params, body, callback) {
     var headers = s3.createXAmzHeaders(params);
     var MAX_CHUNK_SIZE = 4 * 1024;
     var bodyIndex = 0;
     var getNextChunk = function() {
         var ret;
         if (bodyIndex >= dataSize) {
             ret = null;
         } else {
             ret = body.slice(bodyIndex, bodyIndex + MAX_CHUNK_SIZE);
         }
         if (ret) {
             bodyIndex += ret.length || ret.byteLength;
         }
         return ret;
     }
     s3.put(path, headers, getNextChunk, callback);
 }

 putObject(path, params, buffer, callback);

Example 2: Put Data from Storage to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", JSON.stringify(err));
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Store data on the filesystem, then send that stored data.
 */

var path = "/path/to/storage.txt";
var dataSize = 16384;
var params = {
    "ContentLength": dataSize.toString(),
}

var store = storage.open();
var head = "HEADER:";

// Create a test data string and save it to memory
var s = "";
// Up to 4KB of data, to be compatible with put()
while(s.length < 1024) {
    s += "testdata";
}
for (var i = 0; i < 24; i++) {
    store.writeFrame(head, s);
}

// If there was an error during memory save, recover it.
if (!store.readFrame(head, 0)) {
    var ret = store.recoverFrame(head);
}

var putStringFromStorage = function (strStore) {
    var headers = s3.createXAmzHeaders(params);
    var getData;
    if (strStore) {
        if (strStore.getReadableSize() < dataSize) {
            return null;
        }
        if (head) {
            // Get total amount of data in frames with this head
            var frames = strStore.searchAllFrame(head, 0);
            if (frames < dataSize) { // not enough stored data
                return null;
            }
            // Get total number of frames with this head
            frames = strStore.searchAllFrame(head, 1);
            var frameIndex = -1;
            var frameDataCollectedSize = 0;
            var getData = function() {
                if (frameIndex++ < frames && frameDataCollectedSize < dataSize) {
                    var ret = strStore.searchFrame(head, frameIndex);
                    frameDataCollectedSize += ret.length;
                    return ret;
                } else {
                    return null;
                }
            }
        }
    }
    s3.put(path, headers, getData, callback);
}
putStringFromStorage(store);

Example 3: Put Custom Data to S3

// IMPORTED LIBRARIES
// - AWS_S3_V2

// Logging setup
log.setLevel(2); //-1:NONE 0:ERROR 1:WARNING 2:DEBUG 3:TRACE
log.printLevel(2); //0:DISABLE 1:LOG 2:CONSOLE 3:BOTH
log.clear();

// Set config
var config = {
    region: "<string>",
    accessKeyId: "<string>",
    secretKey: "<string>",
    bucket: "<string>",
    ca: awsS3Ca
};

var callback = function (err, data) {
    if (err) {
        print("Error!", JSON.stringify(err));
    } else {
        print(data.statusCode, data.statusMessage);
        print(data.body);
    }
}

var s3 = new AWS_S3(config);

/**
 * Send data to S3 using a custom getData method.
 */

 var path = "/path/to/custom.txt";
 var dataSize = 256;
 var params = {
     "Content-Length": dataSize.toString()
 }

 var send = false;
 var getData = function () { // Must return String, ArrayBuffer, or null.
     if (send) {
         return null;
     } else {
         send = true;
         var i = 0;
         var ret = "";
         while (i++ < dataSize) {
             ret += String.fromCharCode((i % 122) + 32); // ASCII table
         }
         return ret;
     }
 }

 s3.put(path, params, getData, callback);

The company names and product names mentioned above are registered trademarks or trademarks of their respective companies.

---

Updated: 2020/08/27