Approaches on ingesting IoT data from cloud gateway

Question

I would like to hear your insights about an IoT data ingesting case. In AWS IoT hub, thing shadows are virtual representation of physical ones. What i understood from the figure below is whenever a thing sends a data to platform via a message broker, thing shadows and rule engine portions get the same sensor data concurrently and process it.

Are my conclusions correct ?

• Things shadow system is subscribed to message broker and gets sensor data, updates their shadow actors. Shadow side is also responsible for storing sensor data such an event sourcing mechanism.
• The thing shadow system does not perform any rules, it is just for performing event sourcing and keeping last known state in virtual thing actors.
• The same sensor data also is an inbound data to rules engine. Rules engine is just and ECA (event condition action) type system that handle streaming data and decides what it will do with them. This means every incoming data eventually will be processed in rules engine portion.

Answer 1

Below are my comments to your conclusions.

What i understood from the figure below is whenever a thing sends a data to platform via a message broker, thing shadows and rule engine portions get the same sensor data concurrently and process it.

Changes in the thing shadow can trigger an action registered in the rule engine. There are specific topics associated with a thing shadow that you can subscribe the rule engine to, in order to perform one or many action(s) in response.

Things shadow system is subscribed to message broker and gets sensor data, updates their shadow actors. Shadow side is also responsible for storing sensor data such an event sourcing mechanism.

You can update the device shadow by using the REST API, or dedicated MQTT topics to publish on specific shadow topics. The shadow does not constitute an event-sourcing system by itself, but a representation of the data model associated with a physical device, as you said.

You can however create a rule that listens for changes on one or more shadow instances, and register the changes into DynamoDB for instance, in a time-series manner. You'll then have an event-sourcing system allowing you to store the previous states, or changes, sent by a device during an arbitrary amount of time.

The thing shadow system does not perform any rules, it is just for performing event sourcing and keeping last known state in virtual thing actors.

The thing shadow keeps the desired and reported state of a physical device in the cloud. It does not execute rules, but emits messages on MQTT topics when events happen within the shadow. These messages can then be captured by the rules engine to execute actions.

The same sensor data also is an inbound data to rules engine. Rules engine is just and ECA (event condition action) type system that handle streaming data and decides what it will do with them. This means every incoming data eventually will be processed in rules engine portion.

The rules engine does not listen by default on an MQTT topic, and hence, on data sent by devices to the Device Gateway. You must register in the rules engine the topics you'd like to listen to along with their associated actions.

Other than that, the rules engine allows you to describe your rules in ANSI SQL, meaning that you are able to specify the origin of your data (the FROM in your SQL statement), the specific fields in a JSON payload you are interested in capturing (SELECT), and an optional condition specifying on what condition the rule should be triggered (WHERE).

An example of a rule listening on the fictive topic device/+/telemetry and interested in capturing all the fields in the received payload would be :

SELECT * FROM device/+/telemetry

Note how the + can be used as a placeholder for any device identifier for instance.