Stream 100MB+ file from AWS Cloud to the Thing using AWS IoT

Question

So here is the scenario. We have a IoT device/thing which upon setting up need some drivers and firmware. Some files are huge in size like 100MB+ and in those cases instead of sending the data in one go, we would like stream the data from the AWS Cloud to the THING.

I have been looking for ways to do it, and there is an option like IoT Streaming service, but there is no basic code documentation on how to do it in Nodejs. I'm a newbie with nodejs and IoT stuff so if someone can point me towards some help documentation or sites to see how to do it, that would be great!

Using the below code, I cannot transfer more than 10MB.

iot.createStream(params, function (err, data) {
   if (err) return err;
   return data;
})

Error that we are getting:

File size 221413376 of stream file 0 in stream exceeds the file size limit

Params for createStream:

    files: [ /* required */
      {
        fileId: 0,
        s3Location: {
          bucket: bucket, /* required */
          key: params.s3key
        }
      }
    ],
    roleArn: config.jobRoleARN, /* required */
    streamId: uuidv1(), /* required */
   description: 'Get file as Stream'
  }```

Answer 1

This documentation from AWS suggests that the maximum file size for streaming is 24MB. If that's the case, you may need to try and break up the file into smaller pieces or see if it can be compressed to a reasonable size (or both).

Only additional note: depending on what you're doing with the returned data, you might want to utilize pipes and transform/writable streams to take advantage of node's backpressure handling mechanisms, like below:

const request = iot.createStream(params);
const out = fs.createWriteStream('/path/to/outfile.jpg');

request.createReadStream().pipe(out);

When dealing with file I/O alone, it's probably not a big deal because the file output should be faster than the data coming in over the network, but it's still good practice to use pipes where you can and avoid situations where you're listening directly to the 'data' event of an incoming stream.