Docker container communication with other container on diffirent host/server

I am having two servers (CentOS8).
On server1 I have mysql-server container and on server2 I have zabbix-front-end i.e zabbix-web-apache-mysql (container name zabbixfrontend). I am trying to connect to mysql-server from zabbixfrontend container. Getting error

bash-4.4$ mysql -h <MYSQL_SERVER_IP> -P 3306 -uroot -p 
Enter password: 
ERROR 2002 (HY000): Can't connect to MySQL server on '<MYSQL_SERVER_IP>' (115)

When I do nc from zabbixfrontend container to my mysql-server IP I get "No route to host." error message.

bash-4.4$ nc -zv <MYSQL_SERVER_IP> 3306
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: No route to host.

NOTE : I am successfully do nc from the host machine (server2) mysql-server container.

docker-compose.yml

version: '3.5'
services:
 zabbix-web-apache-mysql:
  image: zabbix/zabbix-web-apache-mysql:centos-8.0-latest
  container_name: zabbixfrontend
  #network_mode: host
  ports:
   - "80:8080"
   - "443:8443"
  volumes:
   - /etc/localtime:/etc/localtime:ro
   - /etc/timezone:/etc/timezone:ro
   - ./zbx_env/etc/ssl/apache2:/etc/ssl/apache2:ro
   - ./usr/share/zabbix/:/usr/share/zabbix/
  env_file:
   - .env_db_mysql
   - .env_web
  secrets:
   - MYSQL_USER
   - MYSQL_PASSWORD
   - MYSQL_ROOT_PASSWORD
     # zbx_net_frontend:
  sysctls:
   - net.core.somaxconn=65535

secrets:
  MYSQL_USER:
    file: ./.MYSQL_USER
  MYSQL_PASSWORD:
    file: ./.MYSQL_PASSWORD
  MYSQL_ROOT_PASSWORD:
    file: ./.MYSQL_ROOT_PASSWORD

docker logs zabbixfrontend out as below

** Deploying Zabbix web-interface (Apache) with MySQL database
** Using MYSQL_USER variable from ENV
** Using MYSQL_PASSWORD variable from ENV
********************
* DB_SERVER_HOST: <MYSQL_SERVER_IP>
* DB_SERVER_PORT: 3306
* DB_SERVER_DBNAME: zabbix
********************
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...
**** MySQL server is not available. Waiting 5 seconds...

Solution

The nc message is telling the truth: No route to host.

This happens because when you deploy your front-end container in the docker bridge network, its IP address belongs to the 172.18.0.0/16 subnet and you a are trying to reach an the database via an IP address that belongs to a different subnet (10.0.0.0/16).

On the other hand, when you deploy your front-end container on the host network, you no longer face that problem, because now the IP is literally using the IP address of the host machine, 10.0.0.2 and there is no need for a route to be explicitly created to reach 10.0.0.3.
Now the problem you are facing is that you can no longer access the web-ui via the browser. This happens because I assume you kept the ports:" option in your docker-compose.yml and tried to access the service on localhost:80/443. The source and destination ports do not need to be specified if you run the container on the host network. The container will just listen directly on the host on the port that's opened inside the container.

Try to run the front-end container with this config and then access it on localhost:8080 and localhost:8443:

...  
network_mode: host
#  ports:
#   - "80:8080"
#   - "443:8443"
  volumes:
...

Running containers on the host network is not something that I would usually recommend, but hence your setup is quite special, having one container running on one docker host and another container running in another independent docker host, I assume you don't want create an overlay network and eventually register the two docker hosts to a swarm.