Relabel instance to hostname in Prometheus

Question

I have Prometheus scraping metrics from node exporters on several machines with a config like this:

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets:
        - 1.2.3.4:9100
        - 2.3.4.5:9100
        - 3.4.5.6:9100

When viewed in Grafana, these instances are assigned rather meaningless IP addresses; instead, I would prefer to see their hostnames. I think you should be able to relabel the instance label to match the hostname of a node, so I tried using relabelling rules like this, to no effect whatsoever:

relabel_configs:
  - source_labels: ['nodename']
    target_label: 'instance'

I can manually relabel every target, but that requires hardcoding every hostname into Prometheus, which is not really nice. I see that the node exporter provides the metric node_uname_info that contains the hostname, but how do I extract it from there?

node_uname_info{domainname="(none)",machine="x86_64",nodename="myhostname",release="4.13.0-32-generic",sysname="Linux",version="..."} 1

Answer 1

I just came across this problem and the solution is to use a group_left to resolve this problem. You can't relabel with a nonexistent value in the request, you are limited to the different parameters that you gave to Prometheus or those that exists in the module use for the request (gcp,aws...).

So the solution I used is to combine an existing value containing what we want (the hostnmame) with a metric from the node exporter. Our answer exist inside the node_uname_info metric which contains the nodename value.

I used the answer to this post as a model for my request: https://stackoverflow.com/a/50357418 .

The solution is this one:

node_memory_Active_bytes
  * on(instance) group_left(nodename)
node_uname_info

With this, the node_memory_Active_bytes metric which contains only instance and job labels by default, gets an additional nodename label that you can use in the description field of Grafana.

Hope that this will help others.