Search code examples
dockercontainersprometheus

Prometheus target down at localhost:9323 for docker matrices

I am running Prometheus inside docker container. I want to track metrics of docker. My docker command to run Prometheus is,

docker run -d --name prometheus \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml  -p 9090:9090\
    prom/prometheus

My prometheus.yml file contains.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  external_labels:
      monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'


    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'

    static_configs:
      - targets: ['localhost:9323']

And I have updated daemon.json for docker too. It contains,

{
  "data-root": "/docker-root2/docker",
  "metrics-addr" : "127.0.0.1:9323",
  "experimental": true
}

And I also tried to change it to,

{
  "data-root": "/docker-root2/docker",
  "metrics-addr" : "0.0.0.0:9323",
  "experimental": true
}

I have changed docker root directory to "data-root": "/docker-root2/docker", This doesn't have any connection with question.

Followed docker official tutorial link but this is not working for me. But still I am getting status Down for target docker

Get "http://localhost:9323/metrics": dial tcp 127.0.0.1:9323: connect: connection refused

I want to get the expected result of target status to UP. Further I want to get docker matrices.

Solution

  • I have solved this problem by changing two things. I was running Prometheus inside container, so it doesn't have access to the local machine through localhost. For that I updated my prometheus.yml file and changed,

    static_configs:
      - targets: ['localhost:9323']

    to my host(system) IP address.

    static_configs:
      - targets: ['192.168.8.123:9323']

    Second thing, I recreated container with an extra option --add-host:

    docker run -d --name prometheus \
    --add-host host.docker.internal:host-gateway \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
    -p 9090:9090 \
    prom/prometheus