Converting time series CSV to Prometheus's OpenMetrics

Question

I have a time series dataset in the following format:

time,id_station,s0,s1,s2,s3
2019-03-01T00:00:00,st0,0.093684,0.601416,0.020954,0.179308
2019-03-01T00:00:10,st0,0.181069,0.323754,0.624349,0.043567
...

I need to convert this csv file to OpenMetrics format to be able to load it to Prometheus. I tried a couple of tools on GitHub such as csv-exporter but it did not work for me.

How could I make the conversion of this csv dataset to OpenMetrics format?

Answer 1

You can convert metrics your self.

Here is an example of Python a function: it will split csv string of your format and generate metrics based on it.

from typing import List
import datetime

METRIC_COMMON_PREFIX = 'my_sensor_data_s'


def get_metrics(csvline: str) -> List[str]:
    result = []
    values = csvline.split(',')
    my_timestamp = int(datetime.datetime.fromisoformat(values[0]).timestamp())

    for i in range(2, len(values)):
        result.append(f'{METRIC_COMMON_PREFIX}{i - 1}{{id_station="{values[1]}"}} {values[i]} {my_timestamp}')

    return result


print('\n'.join(get_metrics("2019-03-01T00:00:00,st0,0.093684,0.601416,0.020954,0.179308")))
print('\n'.join(get_metrics('2019-03-02T00:00:10,st0,0.181069,0.323754,0.624349,0.043567')))

Example of output:

my_sensor_data_s1{id_station="st0"} 0.093684 1551391200
my_sensor_data_s2{id_station="st0"} 0.601416 1551391200
my_sensor_data_s3{id_station="st0"} 0.020954 1551391200
my_sensor_data_s4{id_station="st0"} 0.179308 1551391200
my_sensor_data_s1{id_station="st0"} 0.181069 1551477610
my_sensor_data_s2{id_station="st0"} 0.323754 1551477610
my_sensor_data_s3{id_station="st0"} 0.624349 1551477610
my_sensor_data_s4{id_station="st0"} 0.043567 1551477610

Here assumed that your columns after second are values of different sensors, so a different metric is generated for every column.

---

To convert whole file you can employ script like this:

with open('in.csv') as infile:
    with open('out.prom', 'w') as outfile:
        for line in infile:
            metrics = get_metrics(line.rstrip())
            for metric in metrics:
                outfile.write(metric + '\n')