Search code examples
time-seriesprometheusinfluxdb

Writing the data to the timeseries database over unstable network


I'm trying to find a time series database for the following scenario:

  1. Some sensor on raspberry pi provides the realtime data.
  2. Some application takes the data and pushes to the time series database.
  3. If network is off (GSM modem ran out of money or rain or something else), store data locally.
  4. Once network is available the data should be synchronised to the time series database in the cloud. So no missing data and no duplicates.
  5. (Optionally) query database from Grafana

I'm looking for time series database that can handle 3. and 4. for me. Is there any?

I can start Prometheus in federated mode (Can I?) and keep one node on raspberry pi for initial ingestion and another node in the cloud for collecting the data. But that setup would instantly consume 64mb+ of memory for Prometheus node.


Solution

  • Take a look at vmagent. It can be installed at every device where metrics from local sensors must be collected (e.g. at the edge), and collect all these metrics via various popular data ingestion protocols. Then it can push the collected metrics to a centralized time series database such as VictoriaMetrics. Vmagent buffers the collected metrics on the local storage when the connection to a centralized database is unavailable, and pushes the buffered data to the database as soon as the connection is recovered. Vmagent works on Rasberry PI and on any device with ARM, ARM64 or AMD64 architecture.

    See use cases for vmagent for more details.