Prometheus: How to replace Consul server ports with regex?

Question

When Prometheus uses Consul's auto-discovery feature to get the list of targets to monitor, it also gets Consul servers themselves. This is great - we want to monitor these guys with Prometheus. The problem is that Consul reports these nodes with port 8300, which is not the port we use to monitor targets.

How does one replace ports received from Consul to something else? I know this is possible with Prometheus relabel_configs but I've yet to successfully configure it.

Answer 1

I've eventually figured it out. below is a working example. As the documentation specifies, the address keyword might not work for all setups - you might want to try "<__meta_consul_address>:<__meta_consul_service_port>" instead.

   - source_labels: ['__address__']
     separator:     ':'
     regex:         '(.*):(8300)'
     target_label:  '__address__'
     replacement:   '${1}:9126'