I'm currently facing an issue where the counter metric values in my application reset to their default values upon application restarts, instead of considering the last saved values in Prometheus TSDB.
I'm exporting custom metrics from my application to Prometheus TSDB using Opentelemetry exporter and Opentelemetry collector. The application successfully exports metric values to Prometheus, and I can view these values in the Prometheus UI. However, upon restarting the application, the metric values reset to their default values and start updating based on these defaults, rather than retaining the last saved values in Prometheus TSDB.
Is there a way to configure the application or Prometheus to ensure that the metric values are updated based on the last saved values in TSDB, even after application restarts?
Here is the sample data:
| API Name | Status | Count |
|---|---|---|
| /api/abc | 200 | 2 |
| /api/abc | 400 | 1 |
| /api/bcd | 200 | 3 |
Counters works well with restart, but you should use the rate function to visualize it. In counters, the most important is the rate of the changes e.g.: how much it went up or down. For other types such as gauges, resetting should not be a thing. In your specific problem, if you'd like to see the total requests you should the function increase, it will give you the approximate request numbers instead of the rate they go up/down.