How can I get cAdvisor and Node Exporter to connect to Prometheus in a docker-compose environment?

Question

I am trying to setup a monitoring system that runs on Traefik reverse proxy by these three components. And at some point I lost the plot and neither cAdvisor nor node-exporter are recognized targets by Prometheus.

Both of them are No Data or N/A when I set Grafana dashboards of both services.

Both of them are reachable from<container-ip>:<port> but unreachable from localhost:<port> or <container-name>:<port>

example:

http://172.26.0.9:9100/metrics : Functioning service
http://localhost:9100/metrics  : CONNECTION_REFUSED
http://localhost:9100/metrics  : Infinite loading

http://localhost:9090/api/v1/targets:

{
  "status": "success",
  "data": {
    "activeTargets": [
      {
        "discoveredLabels": {
          "__address__": "localhost:9090",
          "__metrics_path__": "/metrics",
          "__scheme__": "http",
          "__scrape_interval__": "15s",
          "__scrape_timeout__": "10s",
          "job": "prometheus"
        },
        "labels": {
          "instance": "localhost:9090",
          "job": "prometheus"
        },
        "scrapePool": "prometheus",
        "scrapeUrl": "http://localhost:9090/metrics",
        "globalUrl": "http://336decec6cb5:9090/metrics",
        "lastError": "",
        "lastScrape": "2023-12-13T19:23:44.751672909Z",
        "lastScrapeDuration": 0.003435381,
        "health": "up",
        "scrapeInterval": "15s",
        "scrapeTimeout": "10s"
      }
    ],
    "droppedTargets": [],
    "droppedTargetCounts": {
      "prometheus": 0
    }
  }
}

this is the docker-compose.yml:

version: "3"

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.10
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock
    container_name: traefik

    #Snip
    #Snip
    #Snip

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus.yml
    container_name: prometheus

  node-exporter:
    image: quay.io/prometheus/node-exporter
    container_name: node-exporter

  cadvisor:
    image: google/cadvisor:latest
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    command:
      - "-port=8100"
    container_name: cadvisor

Here is my prometheus.yml:

global:
  scrape_interval: 15s

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100'] 

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8100'] 

What I have tried so far as replacement for cadvisor:8100 and node-exporter:9100:

localhost:<port>
${docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'}:<port>
host.docker.internal

One thing to note may be; before using docker-compose but containerizing with docker commands in official documentation, I could connect with the second replacement.

Maybe another clue:

$ docker logs cadvisor
W1213 19:20:01.331627       1 container.go:409] Failed to create summary reader for "/snap.snap-store.ubuntu-software": none of the resources are being tracked.

Answer 1

Two things:

1. prometheus.yml is mounted incorrectly:

Should be mounted into /etc/prometheus/prometheus.yml:

volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml

2. prometheus.yml is missing scrape_configs

Should be:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100'] 

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8100'] 

3. Optional: cadvisor registry change

Should be:

gcr.io/cadvisor/cadvisor:v0.36.0