prometheus cannot reach node exporter

Question

I am learning prometheus and node_exporter and trying to set up them locally by docker. I downloaded docker images from this https://hub.docker.com/r/prom/node-exporter and this https://hub.docker.com/r/prom/prometheus, then run them with commands:

docker run -d -p 9100:9100 prom/node-exporter

docker run \
    -d \
    -p 9090:9090 \
    -v /Users/person/sandbox/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

before running prometheus I added yml file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
- job_name: 'node'
  static_configs:
  - targets: ['localhost:9100']

But it seems that prometheus can not reach node_export. I see metrics from node_export ->

and I see metrics from prometheus ->

but I cannot get node_export metrics from prometheus web-browser from:

By the way prometheus read yml file:

What did I wrong?

UDP: nice, it seems that localhost:9100 is unreachable

Answer 1

This is happening because inside the Prometheus container, localhost is being resolved to the container's loopback network interface. To mitigate this, use host networking via explicitly specifying --network=host when starting these containers. When using host networking, Docker won't allocate container's their own networking namespace. Rather they'll share it from their host.

Alternatively, a better approach is to create an overlay network.

docker network create prom
docker network connect prom <prometheus_container_name>
docker network connect prom <node_exporter_container_name>

After creating and connecting containers to the overlay network, they should be accessible from one another by their Docker name ¹.