Search code examples
google-cloud-platformgoogle-cloud-iot

Where does data received from an IOT device get stored on Google Cloud?

I started playing with Google Cloud IOT Core today. I went through the tutorial and ran a script to send messages from the virtual device back to my topic (if this is the correct phrasing). If I want to pull a message sent by my virtual device, I run the following:

gcloud pubsub subscriptions pull --auto-ack projects/glass-potion-314513/subscriptions/my-subscription

And the following is returned:

┌──────────────────────────────────────┬──────────────────┬──────────────┬────────────────────────────────────┬──────────────────┐
│                 DATA                 │    MESSAGE_ID    │ ORDERING_KEY │             ATTRIBUTES             │ DELIVERY_ATTEMPT │
├──────────────────────────────────────┼──────────────────┼──────────────┼────────────────────────────────────┼──────────────────┤
│ my-registry/my-node-device-payload-3 │ 2426853517884309 │              │ deviceId=my-node-device            │                  │
│                                      │                  │              │ deviceNumId=2812664419074966       │                  │
│                                      │                  │              │ deviceRegistryId=my-registry       │                  │
│                                      │                  │              │ deviceRegistryLocation=us-central1 │                  │
│                                      │                  │              │ projectId=glass-potion-314513      │                  │
│                                      │                  │              │ subFolder=                         │                  │
└──────────────────────────────────────┴──────────────────┴──────────────┴────────────────────────────────────┴──────────────────┘

So everything went well.

However, where is this data actually stored and how can I access for use in downstream processes, such as analytics, etc? I've explored the Cloud IOT UI but can't find anything there.

Solution

  • Cloud IOT does not dictate you the way your devices data is stored.

    Please, consider review the IoT technical overview in the Google Cloud Architecture center and some of the proposed architectures, especially Designing a Connected Vehicle Platform on IoT Core.

    The IoT Core main documentation provides a very handy diagram as well:

    enter image description here

    All these architectures are generally based on the processing of the information received from your devices in Pub/Sub in some or other way.

    Pub/Sub is a message broker, similar - but with subtle differences - to other systems like Kafka, Active MQ, etcetera. It allows you to decouple and at the same interconnect different services in the Google Cloud Platform (different services can publish information to a certain topic, while other can consume it). But it will not provide a way for message persistent storage on its own: by default, the message retention period is at most 7 seven days.

    The responsibility for storing those messages if required lies with the services consuming those messages.

    You can use any service that can consume messages from those Pub/Sub topics, but typically you will use products like Dataflow, an advanced stream or batch data processing tool, that will allow you to process your device data and to distribute it to different storage services if necessary.

    If your are interested in analytics, the way to go should be either Bigtable or better BigQuery. There are several Dataflow templates available out of the box for storing your Pub/Sub information in BigQuery. Please, consider review this excellent related article as well.

    You can store your device information in other services as well, like Cloud SQL, managed relational database services versions of MySQL, PostgreSQL and SQL Server, Cloud Firestore (formerly Datastore), a flavor or NoSQL database, or even Cloud Storage, for storing your device information as plain, unstructured objects. Although a bit dated now, please, see the section What Storage type? in the extraordinary grumpygrace.dev flowcharts page for more information about the different storage options available.