I am going to explain the project I am currently working on and please bear with me since English is not my first language.
The project I am working on involves a series of devices connected to the service Watson IOT that periodically (about once every second, not a constant refresh rate though) send data. The data I am talking about comes from many sensors that are very different from each other and are all in the same device. Since they are different, some send data very frequently and some others very rarely. For example I have a temperature sensor that sends data pretty much every second and a 'fault' sensor that sends data only if something goes wrong.
{
"deviceId": "Oven0000",
"data": {
"h": 41.6,
"utctime": 1502222544044
}
}
As you can see from the example above, I just have a "h" parameter and a "utctime" that are saved, not the whole device state.
{
"deviceId": "Oven0000",
"data": {
"utctime": 1502269142953,
"A": 17,
"F": true,
"G": false,
"I": true,
"J": true,
"K": true,
"W": false,
"a": 11,
"b": 4,
"c": 0,
"e": false,
"f": true,
"g": 14,
"h": 60.3,
"q": 1,
"r": 1,
"t": 1,
"x": true,
"z": true
}
}
This example is a full state where all the data is added.
All of this info is then sent from Watson IOT to a Cloudant instance in which it is automatically saved in month indexed databases. Right now I just have a back-end that is responsible for taking all docs from Cloudant, adding to each doc the whole device state, and sending it to a frontend which then visualizes them (see example above).
The backend is written using NodeJS, whilst the frontend is written using Vue.js and some graphical libraries.
Right now, I would like to implement some sort of data analysis on my docs, but not the ones that are stored in the Cloudant DBs since they only have single values. Rather, I'd like to conduct an analysis of the ones that the backend creates, the ones with the full state of the device.
I have thought of having a secondary database, maybe a relational one, where I can send my parsed data from the backend and make some queries.
What do you think of this approach? Is there a better solution? Would you recommend some other types of DBMS?
Have you considered using the Data Management capabilities of the Watson IoT Platform to maintain the state of your device for you? This functionality allows you to define the structure of the state using JSON schema and define mappings that will aggregate the data from different inbound events into a single view of the state.