Search code examples
dockergrafanadocker-swarm

Grafana Mimir Service Discovery over DNS yields no results

I'm running grafana mimir in docker and tried clustering it with the members being discovered by a DNS query, but unfortunately, it keeps telling me

caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=tasks.monitoring_mimir err="lookup tasks.monitoring_mimir on 127.0.0.11:53: no such host"
caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=tasks.monitoring_mimir

dig query

As you can see on the screenshot, the query returns the correct IPs (they are pingable and checking the network shows that those are, in fact, the right ones) and also uses the docker dns server running on 127.0.0.11:53

Also, when running the dig query as the entrypoint, it still resolves correctly, therefore I doubt that it only works because it takes me some time to execute the query

The configs used can be found in this gist, I deploy the stack with docker deploy --compose-file docker-compose.yml monitoring

Solution

  • Unfortunately, this seems to still be an issue to this day.

    However, there is a workaround: manually overriding the IP used for discovery