Why is my AWS IoT SDK thing shadow update request timing out using the Node SDK?

Question

Following an AWS example here and referencing a balena.io example, I'm attempting to to get a "thing" (currently a script on my Mac) to update a thing shadow on AWS.

I'm getting close. So far I can successfully register interest in a thing shadow (UPDATE: and subscribe and publish to an MQTT topic, receiving updtes). However, I'm getting a timeout when attempting to update the shadow. Originally, I was running into a timeout when registering interest due to a missing policy on the thing's certificate, now a basic one in place. My current thought is that maybe I need to use a different root CA cert (currently using CA1 provided) or maybe something is wrong with my base64 encoded cert strings, encoded with:

openssl base64 -in some-cert.pem -out some-cert.txt
#gets copied to clipboard and pasted in UI env field
pbcopy < some-cert.txt

Here's what I have so far. Please mind the TODO notes as this is a work-in-progress.

Policy (over-permissive for now):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}

troubleshooting.ts

// application endpoint
const AWS_ENDPOINT = "my-endpoint.iot.us-east-1.amazonaws.com";
//  thing private key encoded in base64
const AWS_PRIVATE_CERT = "gets set in balena.io UI as base64 string";
// aws root CA1 certificate encoded in base64
const AWS_ROOT_CERT = "gets set in balena.io UI as base64 string";
// thing certificate
const AWS_THING_CERT = "gets set in balena.io UI as base64 string";
const AWS_REGION = 'us-east-1';

import { DataUploader } from './data-uploader';
import { DataUploaderOptions } from './model/data-uploader-options';

const dataUploaderOptions: DataUploaderOptions = {
  awsEndpoint: AWS_ENDPOINT,
  awsPrivateCert: AWS_PRIVATE_CERT,
  awsCaCert: AWS_ROOT_CERT,
  awsThingCert: AWS_THING_CERT,
  awsRegion: AWS_REGION
}

const dataUploader = new DataUploader(dataUploaderOptions);

data-uploader.ts

/**
 * Upload image and meta data to AWS as an AWS IoT thing.
 */

import { thingShadow, ThingShadowOptions } from 'aws-iot-device-sdk';
import { DataUploaderOptions } from './model/data-uploader-options';
import { get } from 'lodash';

export class DataUploader {

  // Provided options via constructor parameter
  protected readonly options: DataUploaderOptions;

  // AWS IoT thing shadows
  protected thingShadows: thingShadow;
  protected thingName = 'TwinTigerSecurityCamera';
  protected currentTimeout: NodeJS.Timer;
  // TODO: Typescript wants MqttClient type, better than 'any', check if it makes sense to import here just for type reference.
  protected clientToken: string | void | any;

  // TEMP: Move to class options:
  protected operationTimeout = 10000;
  protected operationTimeoutRetry = 1000;

  constructor(options: DataUploaderOptions) {
    // Set options based on input or defaults if not provided.
    this.options = {
      awsEndpoint: get(options, 'awsEndpoint', ''), // empty will throw an error
      awsPrivateCert: get(options, 'awsPrivateCert', ''), // empty will throw an error
      awsCaCert: get(options, 'awsCaCert', ''), // empty will throw an error
      awsThingCert: get(options, 'awsThingCert', ''), // empty will throw an error
      awsRegion: get(options, 'awsRegion', ''), // empty will throw an error
      awsMqttClientId: get(options, 'awsMqttClientId', this.createUniqueClientId())
    };

    // Proceed no further if AWS options are not set properly.
    if (!this.options.awsEndpoint ||
        !this.options.awsPrivateCert ||
        !this.options.awsCaCert ||
        !this.options.awsThingCert ||
        !this.options.awsRegion) {
      throw new Error('DataUploader constructor: AWS IoT options are required.');
    }

    // setup thingShadow and events
    this.initThingShadow();
  }

  /**
   * 
   */
  protected initThingShadow = () => {
    // create thing shadow: extends IoT 'device' with extra status that can be
    // set/received online or offline (can sync when online)
    const thingShadowOptions: ThingShadowOptions = {
      // TODO: revise 'ca' to 'rootCa1' to help clarify
      clientCert: Buffer.from(this.options.awsThingCert, 'base64'),
      caCert: Buffer.from(this.options.awsCaCert, 'base64'),
      privateKey: Buffer.from(this.options.awsPrivateCert, 'base64'),
      clientId: this.options.awsMqttClientId,
      host: this.options.awsEndpoint,
      region: this.options.awsRegion
    };
    this.thingShadows = new thingShadow(thingShadowOptions);

    this.thingShadows.on('connect', () => {
      console.log('connected');
      this.registerThingShadowInterest();
    });

    // Report the status of update(), get(), and delete() calls.
    this.thingShadows.on('status', (thingName, stat, clientToken, stateObject) => {
      const tmpObj = JSON.stringify(stateObject);
      console.log(`received ${stat} on ${thingName}: ${tmpObj}`);
    });

    this.thingShadows.on('message', (topic, message) => {
      const tmpObj = JSON.stringify(message); 
      console.log(`message received for ${topic}: ${tmpObj}`);
    });

    this.thingShadows.on('foreignStateChange', (thingName, operation, stateObject) => {
      const tmpObj = JSON.stringify(stateObject); 
      console.log(`foreignStateChange happened for ${thingName}, ${operation}: ${tmpObj}`);
    });

    this.thingShadows.on('delta', (thingName, stateObject) => {
      const tmpObj = JSON.stringify(stateObject); 
      console.log(`received delta on ${thingName}: ${tmpObj}`);
    });

    this.thingShadows.on('timeout', (thingName, clientToken) => {
      console.log(`timeout for ${thingName}: ${clientToken}`);
    });
  }

  /**
   * 
   */
  protected setInitialThingShadowState = (): void => {
    // TODO: consider making interface for this
    const cameraState = JSON.stringify({
      state: {
        desired: {
          signedUrlRequests: 10,
          signedUrls: [],
          startedAt: new Date(),
          uploadCount: 0,
          uploadSpeed: 0
        }
      }
    });

    this.thingShadowOperation('update', cameraState);
  }

  /**
   * 
   */
  protected registerThingShadowInterest = () => {
    this.thingShadows.register(this.thingName, { 
      ignoreDeltas: true
    },
    (err, failedTopics) => {
      if (!err && !failedTopics) {
        console.log(`${this.thingName} interest is registered.`);
        this.setInitialThingShadowState();
      } else {
        // TODO: What do we do now? Throw an error?
        const failedString = JSON.stringify(failedTopics);
        console.error(`registerThingShadowInterest error occurred: ${err.message}, failed topics: ${failedString}`);
      }
    });
  }

  /**
   * Thanks: https://github.com/aws/aws-iot-device-sdk-js/blob/master/examples/thing-example.js
   */
  protected thingShadowOperation = (operation: string, state: Object) => {
    // TODO: Check if there's a better way to do this. We want to accept operation
    // parameter as string only (no any), then ensure it's a key of the thingShadow
    // class. It works in TypeScipt, however calling class methods dynamically seems
    // like one of the few cases when the developer wants to ditch TypeScript.
    const operationKey: ('register' | 'unregister' | 'update' | 'get' | 'delete' | 'publish' | 'subscribe' | 'unsubscribe') = <any>operation;
    const clientToken = this.thingShadows[operationKey](this.thingName, state);

    if (clientToken === null) {
       // The thing shadow operation can't be performed because another one
       // is pending. If no other operation is pending, reschedule it after an 
       // interval which is greater than the thing shadow operation timeout.
       if (this.currentTimeout !== null) {
          console.log('Operation in progress, scheduling retry...');
          this.currentTimeout = setTimeout(
             function() {
                this.thingShadowOperation(operation, state);
             },
             this.operationTimeout + this.operationTimeoutRetry);
       }
    } else {
       // Save the client token so that we know when the operation completes.
       this.clientToken = clientToken;
    }
 }

  /**
   * Generate a unique MQTT client id so not to collide with other ids in use.
   */
  // TODO
  createUniqueClientId = (): string => {
    return 'temporaryClientIdWillBeMoreUniqueInTheFuture';
  }
}

model/data-uploader-options.ts

// DataUploader options
export interface DataUploaderOptions {
  // AWS IoT endpoint found in IoT settings in management console
  awsEndpoint: string;

  // AWS IoT private certificate for single device
  awsPrivateCert: string;

  // AWS IoT CA certificate
  awsCaCert: string;

  // AWS IoT thing certificate for single device
  awsThingCert: string;

  // AWS IoT region where thing settings live
  awsRegion: string;

  // an MQTT client id that needs to be unique amongst all other client ids
  awsMqttClientId?: string;
}

What am I missing to update the thing shadow?

---

Update:

When I use the certificates directly and not base64 string versions, I get the same timeout results. Example:

    const thingShadowOptions = {
      keyPath: '/Users/me/Downloads/private-key-aaaaaaaaaa-private.pem.key',
      certPath: '/Users/me/Downloads/thing-aaaaaa-certificate.pem.crt',
      caPath: '/Users/me/Downloads/ca-AmazonRootCA1.pem',
      clientId: this.options.awsMqttClientId,
      host: this.options.awsEndpoint,
      region: this.options.awsRegion
    }

Also, I am able to subscribe to an MQTT topic and publish to it:

    this.thingShadows.on('connect', () => {
      console.log('connected');
      // this.registerThingShadowInterest();

      // TEMP
      this.thingShadows.subscribe('topic_1');
      this.thingShadows.publish('topic_1', JSON.stringify({ test_data: 1}));
    });

Outputs:

message received for topic_1: {"type":"Buffer","data":[123,34,116,101,115,116,95,100,97,116,97,34,58,49,125]}

Answer 1

Debugging the Shadow Update

You can subscribe to the reserved topic $aws/things/+/shadow/# to debug the problem.

This shows a 400 error and a message.

Fixing the Update Payload

The error message is:

"message": "Missing required node: state"

This is visible in the stringified update payload. But the stateObject parameter that is passed to thingShadow.update() should be an object and not a string.

So remove the JSON.stringify from:

const cameraState = JSON.stringify({
  state: {
    desired: {
      signedUrlRequests: 10,
      signedUrls: [],
      startedAt: new Date(),
      uploadCount: 0,
      uploadSpeed: 0
    }
  }
});

and change this to an object:

const cameraState = {
  state: {
    desired: {
      signedUrlRequests: 10,
      signedUrls: [],
      startedAt: new Date(),
      uploadCount: 0,
      uploadSpeed: 0
    }
  }
};

See the documentation at https://github.com/aws/aws-iot-device-sdk-js#update

awsIot.thingShadow#update(thingName, stateObject)

Update the Thing Shadow named thingName with the state specified in the JavaScript object stateObject.

Viewing the Results

The update is visible in the thing's shadow after fixing the parameter format: