Docker swarm can't reach services in another host

Question

Consider the following docker stack

version: "3.2"

services:
  web1:
    image: nginx
    ports:
      - "8081:80"
    deploy:    
      placement:
        constraints:
          - node.role == manager

  web2:
    image: nginx
    ports:
      - "8082:80"

I can easily access both services using my browser with the addresses (http://docker_swarm_manager:8081 and http://docker_swarm_node:8082)

But service web1 cannot access web2 and vice versa.

Ping works fine so DNS is working

root@dfb7b6a65a2e:/# ping web
ping: web: Temporary failure in name resolution
root@dfb7b6a65a2e:/# ping web1
PING web1 (10.0.5.5) 56(84) bytes of data.
64 bytes from 10.0.5.5 (10.0.5.5): icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from 10.0.5.5 (10.0.5.5): icmp_seq=2 ttl=64 time=0.082 ms
...
--- web1 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6009ms
rtt min/avg/max/mdev = 0.067/0.073/0.082/0.005 ms

But accessing the service does not work

root@dfb7b6a65a2e:/# curl -v web2:80
*   Trying 10.0.5.2:80...
* connect to 10.0.5.2 port 80 failed: Connection timed out
* Failed to connect to web2 port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to web2 port 80: Connection timed out

There is no IP/network conflict, docker swarm uses default network, the hosts are using 172.X.X.X/24 network.

Answer 1

Docker swarm is running on a virtual machine (vmware) and vmware use the same port as docker for

From vmware docs

Starting with NSX 6.2.3, the default VXLAN port is 4789, the standard port assigned by IANA.

So the solution is to recreate the docker swarm cluster using a different port

docker swarm init --data-path-port=7788

From docker swarm docs

Port 4789 UDP (configurable) for the container ingress network.