Prometheus metrics end point is running but the target status is down

Question

I have the Prometheus Polkadot metrics endpoint running at http://localhost:9615/metrics I have defined in the config prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "substrate_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9615"]

I got the following error in the Targets

Get "http://localhost:9615/metrics": dial tcp 127.0.0.1:9615: connect: connection refused

My Prometheus is running inside a docker container

docker run -d --name prometheus -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

How to resolve that error?

Answer 1

In order to communicate between two containers, you'll want them to be in the same Docker bridge network. Otherwise, one container's version of localhost is completely isolated from another.

To fix this, try running both containers in the same docker-compose.yaml using the container name instead of localhost.

By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

version: '2'

services:
  polkadot:
    container_name: polkadot
    image: parity/polkadot
    ports:
      - 30333:30333 # p2p port
      - 9933:9933 # rpc port
      - 9944:9944 # ws port
      - 9615:9615
    command: [
      "--name", "PolkaDocker",
      "--ws-external",
      "--rpc-external",
      "--rpc-cors", "all",
      "--prometheus-external"
    ]

  prometheus:
    container_name: prometheus
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command: [
      "--config.file=/etc/prometheus/prometheus.yml"
    ]

Simiarly you'll need to update the host you hit in the prometheus config like so:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["prometheus:9090"] # NOTE HOW ITS NOT LOCALHOST HERE
  - job_name: "substrate_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["polkadot:9615"] # NOTE HOW ITS NOT LOCALHOST HERE

You can find a plethora of resources on the intricacies of Docker networking right in their docs here. But the above configs should get you up and running since both targets came up in Prometheus on my end.