Search code examples
databasenosqltime-seriesdata-modelinggriddb

Possible record collisions due to the size / definition of the TIMESTAMP field in the TIME SERIES type containers of GridDB

I am working with GridDB and I have observed loss of records during the insertions that I attribute to the lack of definition of the timestamp field.

I tried to give more definition in the entry field but saving it makes it trim. Logs do not indicate any data loss or erroneous writing.

A query DB:

[{
"columns":[
  {"name":"original_timestamp","type":"TIMESTAMP"},
  {"name":"FIELD_A","type":"STRING"}
  ...
  {"name":"FIELD_Z","type":"STRING"}
  {"name":"code_timestamp","type":"STRING"}],
  "results":[
  "2019-07-19T11:28:42.328Z",
  "SOME String Value for A",
  ...
  "SOME String Value for Z",
  "2019-07-19 11:28:59.239922"}
]

The number of registered ingested its lower than expected. We're working on a model based on two indexes. Any other idea and / or helpful experience?

Thanks in advance!

Solution

  • GridDB stores TIMESTAMP values in millisecond resolution, inserting records with greater resolution such as micro or nanosecond resolution will result in the timestamp value being truncated. There are three ways to get around the timestamp collisions:

    1. Use a Collection with a long as your first index. In that long, store a Unix Epoch in micro or nanoseconds as required. You will obviously lose some time series functions and have to manually convert comparison operators to a Unix epoch in your desired resolution.

    2. Use a collection and disable the row key (No @RowKey tag in Java or set the last Boolean in ContainerInfo to False in other languages). This will allow multiple records to have the same "row key value". You can enable a secondary index on this column to ensure queries are still fast. TIMESTAMP and TO_TIMESTAMP_MS functions still work but I'm fairly certain none of the other special timestamp functions will work. When I've had to deal with Timestamp collisions in GridDB, this is the solution I chose.

    3. Detect the collisions before you insert and if there is going to be a collision, write the colliding record into a separate container. Use multi-get/query to query all of the containers.