I am writing some checks for Prometheus, and I would like to find out, which variables I may use in templates. I am using $labels.container, but $labels.service and $labels.instance does not work. For example, I am using instance in prometheus queries, but I am not able to use it in template. How can I find out all labels, or other variables I can use in template? I implemented own exporter written in python that controls which containers in docker swarm keep restarting.
I am new with Prometheus and Alertmanager and documentation does not looks very clear according to me. Thank you.
Labels that are available are the labels that are returned by the alerting query. So if your query looks like:
sum ( something_here ) > 0
you will have no labels available.
If you have
sum by (label_foo)(...) > 0
then you will only have label_foo available.
If you do:
metric_name
Then you'll get all the labels that are available on metric_name metric.
Just execute your alerting query directly in prometheus (making sure query does return something - remove any threshold etc.) and you'll see the labels available.