Search code examples
dockerprometheus

Prometheus failes to reconnect docker daemon on localhost

As I am new to monitoring with prometheus I have installed grafana and prometheus with docker compose on the machine I used for testing and it worked to have the IP of the Host in the prometheus.yml

Now after docker compose down docker compose up the connection is refused.

prometheus.yml has

- job_name: 'docker'
  static_configs:
    - targets: ['host.docker.internal:9323']

Docker daemon.json has

"metrics-addr" : "127.0.0.1:9323"

and

curl http://localhost:9323/metrics

works fine from the console but in Prometheus shows this error.

Get "http://host.docker.internal:9323/metrics": dial tcp: lookup host.docker.internal on 127.0.0.11:53: no such host

How would I corretly connect prometheus from inside a docker container to the docker daemon on the host ?

Solution

  • The issue you experienced seems to be related to network configuration and potential permissions settings within your Docker environment. The solution can be derived from the comments and consists of the following steps:

    1. Modify the prometheus.yml configuration file. Replace the host.docker.internal target under the static_configs section with the DNS name of your host machine. The host.docker.internal hostname might not be resolving correctly, hence the need to use the actual DNS name of your host.
    2. Adjust the metrics settings in the daemon.json file to be unrestricted. This setting might be necessary to bypass potential permission-related issues that might be preventing Prometheus from accessing the Docker daemon metrics.

    After performing these changes, ensure your monitoring setup is functioning as expected. If you encounter further issues, additional debugging steps include inspecting the network your containers are operating in using the docker inspect command and verifying the user executing the docker commands to avoid potential permission-related issues.