Can't make any DNS lookups to "external" DNS servers?

Question

I'm trying to fix a bug in my Docker stack for the past few weeks now but to no avail.

This is the compose file in question (stripped down to the relevant stuff):

version: '3.7'
services:
  php:
    ...
    dns:
      - 1.1.1.1
      - 1.0.0.1
networks:
  default:
    internal: true
      driver: overlay

It deploys just fine until I have to do anything that is requires an "external" DNS request.
Eg. in my container I manually run curl https://www.google.com.
This results in the following error

curl: (6) Could not resolve host: google.com

This is the content of my /etc/resolv.conf inside the container:

search finlaydag33k.nl
nameserver 127.0.0.11
options ndots:0

Running docker inspect -f '{{.HostConfig.DNS}}' container-id results in the following output:

[1.1.1.1 1.0.0.1]

My question is, what am I doing wrong and how can I fix this? It can reach services within the same stack by dns name just fine, just not domains that require and external DNS (like 1.1.1.1 or 8.8.8.8). Nothing is beeing blocked by my firewall.

Answer 1

This is a bit of a "doh" moment.
Apparently (though I could not find this in the docs), a container is only hooked up to the default network, which is an overlay network.
This means that one has to explicitly add a bridge network (one should already be present on your swarm) to this container (and also explicitly hook it to the default network so it can communicate with other containers in the stack).

This was not documented so I assumed it was able to reach the outside already.
After manually adding the default and bridge networks to the container, everything works as expected.

version: '3.7'
services:
  php:
    ...
    networks:
      - default
      - bridge
    dns:
      - 1.1.1.1
      - 1.0.0.1
networks:
  default:
    internal: true
      driver: overlay
  bridge:
    external: true