To monitor the state of some systemd-services, we got a State-timeline visualisation in Grafana looking like this:
Picture of our current grafana visualisation
The query for this is:
sum(node_systemd_unit_state{instance=~"^.*$",state="failed"}) by (name, state) > 0
and it outputs the timeline graph, where each bar has a name looking like: {name="certbot.service", state="failed"}
Now, we want to replace every name string like {name="certbot.service", state="failed"} with only the name certbot.service.
I guess using some kind of RegEx like (?<={name=")(.*)(?=",) could help to match everything between {name=" and , state="failed}"
Where (?<={name=") would match everythin before the name,
then (.*) is to replace every possible service-name and
the end (?=",) will cut everything begining with ",
From googling, I already know, that there is the option "label replace" and also an option to configure something in the scraping settings? But as I'm quite new to this topic, I didn't really understand how to use this functions or if they even fit my problem.
This is, what ChatGPT told me to try:
sum(node_systemd_unit_state{instance=~"^.*$",state="failed"}) by (name, state) > 0
| regex_replace(name, '(?<={name=")(.*)(?=",)', '$1')
But Grafana/Prometheus gives me a parsing error after the pipe-symbol
Can you give me some advise, where I could find further information about my problem or even tell me how to edit the query to just output the wanted name?
I already tried googling my problem. It seems, that there are some similar solutions to what could fit into my usecase but I don't have the knowledge to adapt it to my problem. Of course I asked ChatGPT which couldn't help me either.
While it does not exactly match my original question, we found a solution, that would fit our needs.
See, in Grafana, when creating your visualisation, you got the option to edit your Legend.
We used the option custom and as string {{label_name}}.
By simply replacing that string with {{name}} we just got our service-name.