Search code examples
databaseamazon-web-servicesamazon-dynamodbiotaws-iot

DynamoDB Maps vs Lists for storing IoT Data

I'm trying to store IoT Data from data loggers that can have a variety of sensors attached, below is an example. Each logger sends an MQTT message every 20 seconds

"state": {
    "reported": {
      "batv": 5105,
      "ts": 1614595073655,
      "temp": 20,
      "humidity": 50
    }
  }

My Question is in terms of storing these MQTT messages/readings efficiently in a DynamoDB table, should i store the readings in a Map containing Maps like this. (Note this is currently what I'm doing and when the number of readings gets large, it is very slow to load in AWS DynamoDB console.)

{
  "readings": {
    "ts1614592810955": {
      "battery_level": 5089,
      "temp": 20,
      "humidity": 50
    },
    "ts1614593692395": {
      "battery_level": 5093,
      "temp": 20,
      "humidity": 50
    }
  },
  "serial_number": "TDG_logger_thing"
}

The alternative which I'm leaning towards, is by storing readings in a list

{
  "readings": [
    {
      "batv": 5105,
      "ts": 1614594313407,
      "temp": 20,
      "humidity": 50
    },
     {
      "batv": 5105,
      "ts": 1614594313555,
      "temp": 20,
      "humidity": 50
    }
  ],
  "serial_number": "TDG_Logger_Thing"
}

Anyone with knowledge on DynamoDB or storing IoT data have any suggestions? greatly appreciated

(BTW The flow of data is)

Data Logger -> AWS IoT -> AWS Lambda -> DynamoDB

Solution

  • I would not use a map or a list and split those readings and store them in separate items. With the same partition key like the device id, combined with a sort key for every reading, also including the timestamp. That way you can more easily query for all temp data and with the timestamp in the sort key you could use the query to fetch only the measurements from a specific period.

    so primary key would be: PK[device id] - SK[Measurement type - Data time] : (Attributes per measurement)

    After that you can store whatever data you need for each individual measurement. and you can quickly update and retrieve individual measurements, hope it helps.