Search code examples
nginxnetwork-programmingdockerroutesiptables

Docker network issues with nginx proxy container

I am currently trying to setup a docker based jira and confluence platform proxied by nginx and running into some kind of routing and network problems.

The basic setup consists of three docker containers - the nginx conatainer handles the https requests for specific domain names (e.g. jira.mydomain.com, confluence.mydomain.com) and redirects (proxy_pass) the requests to the specific containers for jira and confluence.

This setup is generally working - I can access the jira instance by opening https://jira.mydomain.com and the confluence instance by opening https://confluence.mydomain.com in my browser.

The problem I am running into becomes visible when logging into the jira: enter image description here

And following the Find-out-more-link to:JIRA Health Checks

The suggested resolutions from the provided JIRA health check link unfortunately did not help me to identify and solve the problem. Instead some exceptions in the log file lead to some more hints on the problem:

2017-06-07 15:04:26,980 http-nio-8080-exec-17 ERROR christian.schlaefcke 904x1078x1 eqafq3 84.141.114.234,172.17.0.7 /rest/applinks/3.0/applicationlinkForm/manifest.json [c.a.a.c.rest.ui.CreateApplicationLinkUIResource] ManifestNotFoundException thrown while retrieving manifest ManifestNotFoundException thrown while retrieving manifest com.atlassian.applinks.spi.manifest.ManifestNotFoundException: java.net.NoRouteToHostException: No route to host (Host unreachable) ... Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

And when I follow the hint from this Atlassian knowledge base article and running this curl statement from inside of the JIRA container:

curl -H "Accept: application/json" https://jira.mydomain.com/rest/applinks/1.0/manifest -v

I finally get this error:

* Trying <PUBLIC_IP>... * connect to <PUBLIC_IP> port 443 failed: No route to host * Failed to connect to jira.mydomain.com port 443: No route to host * Closing connection 0 curl: (7) Failed to connect to jira.mydomain.com port 443: No route to host

EDIT: The external URL jira.mydomain.com can be pinged from inside of the container:

root@c9233dc17588:# ping jira.mydomain.com PING jira.mydomain.com (<PUBLIC_IP>) 56(84) bytes of data. 64 bytes from rs226736.mydomain.com (<PUBLIC_IP>): icmp_seq=1 ttl=64 time=0.082 ms 64 bytes from rs226736.mydomain.com (<PUBLIC_IP>): icmp_seq=2 ttl=64 time=0.138 ms 64 bytes from rs226736.mydomain.com (<PUBLIC_IP>): icmp_seq=3 ttl=64 time=0.181 ms

From outside of the JIRA container (e.g. docker host or other machine) the curl statement works fine!

I have quite a good experience with linux in general but my knowledge about networks, routing and iptables is rather limited. Docker is running the current 17.03.1-ce version in combination with docker compose on a centos 7 system:

~]# uname -a Linux rs226736 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

At the moment I don´t even understand what kind of problem (iptables?, routing, docker?) this actually is and how to debug this :-(

I played around with some iptables and nginx related hints found via google - all without success. Any hint pointing me to the right direction would be very much appreciated.

REQUESTED CONFIGS:

NGINX docker-compose.yml

nginx:
  image: nginx
  container_name: nginx
  ports:
    - 80:80
    - 443:443
  external_links:
    - my_domain-jira
    - my_domain-confluence
  volumes:
    - /opt/docker/logs/nginx:/var/log/nginx
    - ./nginx.conf:/etc/nginx/nginx.conf
    - ./certs/jira.mydomain.com.crt:/etc/ssl/certs/jira.mydomain.com.crt
    - ./certs/jira.mydomain.com.key:/etc/ssl/private/jira.mydomain.com.key
    - ./certs/confluence.mydomain.com.crt:/etc/ssl/certs/confluence.mydomain.com.crt
    - ./certs/confluence.mydomain.com.key:/etc/ssl/private/confluence.mydomain.com.key

JIRA docker-compose.yml (Confluence similar):

jira:
  container_name: my_domain-jira
  build: .
  external_links:
   - postgres
  volumes:
   - ./inst/conf/server.xml:/opt/jira/conf/server.xml
   - ./inst/bin/setenv.sh:/opt/jira/bin/setenv.sh
   - /home/jira:/opt/atlassian-home
   - /opt/docker/logs/jira:/opt/jira/logs
   - /etc/localtime:/etc/localtime:ro

NGINX - nginx.conf

upstream jira {
    server my_domain-jira:8080;
}

# begin jira configuration
server {
    listen 80;
    server_name  jira.mydomain.com;

    client_max_body_size 500M;
    rewrite ^ https://$server_name$request_uri? permanent;
}

server {
    listen       443 ssl;
    server_name  jira.mydomain.com;

    ssl          on;
    ssl_certificate      /etc/ssl/certs/jira.mydomain.com.crt;
    ssl_certificate_key  /etc/ssl/private/jira.mydomain.com.key;

    ssl_session_timeout  5m;

    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK';

    server_tokens off;
    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";

    client_max_body_size 500M;

    location / {
        proxy_pass http://jira/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_cache_bypass $http_upgrade;
    }
}

Ideas (nginx / proxy_pass / upstream) mostly picked up from:

Solution

  • After some discussion with the provider of the virtual server it turned out, that conflicting firewall rules between plesk firewall and iptables caused this problem. After the conflict had been fixed by the provider the container could be accessed.

    This problem is solved now - thank´s to anyone who participated!