I have an annoying problem with Docker and my ovh server (ubuntu xenial). I run a NGINX official docker image (15.x) on my server and I forward some port from my server to my docker image (to host serveral services online).
All HTTP request done from outside (using IPV4 or IPV6) manage to reach NGINX. BUT, HTTPS requests done using IPV4 are not able to reach NGINX (IPV6 HTPPS works like a charm).
I listened to port 443 both on my server and in NGINX docker container with tcdump tcpdump -i any port 443 -s0 -n
Result: I saw IPV6 HTTPS request in both my server and NGINX container. I saw IPV4 https request only on my server but not in the NGINX container. => So it seems, this is not a "opened port" problem, because I managed to see IPV4 requests on my server (and I checked IPTables, 443 is forwarded to docker proxy). I think it's a port forwarding pb between my server and Docker proxy, only for HTTPS request done with IPV6.
Here is my container config:
nginx:
depends_on:
- my_php7
image: nginx:latest
volumes:
- "../:/usr/share/nginx/html:rw"
- "./nginx/var/log:/var/log/nginx:rw"
- "./nginx/nginx.template.conf:/etc/nginx/conf.d/nginx.template:rw"
- "./nginx/sites-available:/etc/nginx/sites-available:rw"
- "./nginx/sites-enabled:/etc/nginx/sites-enabled:rw"
links:
- my_php7:webstack_php
ports:
- "8080:8080"
- "8090:90"
- "8091:91"
- "8083:8083"
- "80:80"
- "443:443"
restart: always
command: /bin/bash -c "cp /etc/nginx/conf.d/nginx.template /etc/nginx/conf.d/nginx.conf && nginx -c /etc/nginx/conf.d/nginx.conf"
I checked also net.ipv6.bindv6only:
#> sysctl net.ipv6.bindv6only
net.ipv6.bindv6only = 0
here are my nat infos:
Proto Recv-Q Send-Q Adresse locale Adresse distante Etat PID/Program name
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 19671/node
tcp 0 0 91.121.101.165:914 0.0.0.0:* LISTEN 1696/openvpn-openss
tcp 0 0 91.121.101.165:915 0.0.0.0:* LISTEN 1724/openvpn-openss
tcp 0 0 0.0.0.0:8084 0.0.0.0:* LISTEN 22633/node
tcp 0 0 91.121.101.165:916 0.0.0.0:* LISTEN 1751/openvpn-openss
tcp 0 0 91.121.101.165:917 0.0.0.0:* LISTEN 1779/openvpn-openss
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 929/named
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 947/vsftpd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 943/sshd
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 929/named
tcp 0 0 127.0.0.1:4040 0.0.0.0:* LISTEN 4765/ngrok
tcp6 0 0 :::8082 :::* LISTEN 12095/docker-proxy
tcp6 0 0 :::8083 :::* LISTEN 12837/docker-proxy
tcp6 0 0 :::2003 :::* LISTEN 12058/docker-proxy
tcp6 0 0 :::9300 :::* LISTEN 12006/docker-proxy
tcp6 0 0 :::85 :::* LISTEN 12070/docker-proxy
tcp6 0 0 ::1:53 :::* LISTEN 929/named
tcp6 0 0 :::22 :::* LISTEN 943/sshd
tcp6 0 0 :::3000 :::* LISTEN 12544/docker-proxy
tcp6 0 0 ::1:953 :::* LISTEN 929/named
tcp6 0 0 :::8090 :::* LISTEN 12861/docker-proxy
tcp6 0 0 :::443 :::* LISTEN 12885/docker-proxy
tcp6 0 0 :::8091 :::* LISTEN 12849/docker-proxy
tcp6 0 0 :::92 :::* LISTEN 12082/docker-proxy
tcp6 0 0 :::4000 :::* LISTEN 12532/docker-proxy
tcp6 0 0 :::9000 :::* LISTEN 12521/docker-proxy
tcp6 0 0 :::5000 :::* LISTEN 12106/docker-proxy
tcp6 0 0 :::3306 :::* LISTEN 12119/docker-proxy
tcp6 0 0 :::80 :::* LISTEN 12897/docker-proxy
tcp6 0 0 :::8080 :::* LISTEN 12873/docker-proxy
tcp6 0 0 :::9200 :::* LISTEN 12035/docker-proxy
tcp6 0 0 :::4400 :::* LISTEN 28319/node
udp6 3072 0 :::40195 :::* 1081/collectd
udp6 0 0 :::8124 :::* 12046/docker-proxy
udp6 0 0 ::1:53 :::* 929/named
udp6 0 0 ff18::efc0:4a42:25826 :::* 1081/collectd
For those who wonder why there is only IPV6 forwarding displayed for 443:
there is the same for 80 port and it s working fine
From https://groups.google.com/d/msg/golang-nuts/F5HE7Eqb6iM/q_um2VqT5vAJ
on linux, by default, net.ipv6.bindv6only is 0, so ipv4 packets could also be received from ipv6 sockets with ipv4-mapped ipv6 address. thus you only need to listen on tcp6 socket and we can support both ipv4 and ipv6.
if you want explicitly only listen on ipv4 port, you will have to use net.Listen("tcp4", "0.0.0.0:3000") and then pass the listener to http.Serve.
I tried to rebuild the container without success. I also tried to open 443 port in another docker => it was not able to get IPV4 HTTPS requests.
To resume:
All HTTP requests (IPV4/IPV6) works
Only IPV6 HTTPS works, IPV4 seems to not beeing forwarded to Docker
IPtables, Docker conf and net stats are ok.
any ideas? :)
[edit]
@leodotcloud
sure here it is (I hide the server name with xxxx)
The vhost config
server {
listen 443 ssl http2;
server_name www.xxx.com;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;
ssl_certificate /etc/letsencrypt/live/xxx.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/xxxx.com/privkey.pem;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_prefer_server_ciphers on;
charset utf-8;
location / {
proxy_pass http://node;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
}
The nginx conf for SSL
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
....
}
[edit 2] I tried to change my docker-compose configuration from "443:443" to "0.0.0.0:443:443" to force ipv4 binding .
And I got this error : Bind for 0.0.0.0:443 failed: port is already allocated
So I have something which use my IPV4 connexion on 443.
When I use the "443:443" syntax to bind IPV6 and IPV4, docker bind IPV6 but don't check IPV4 (maybe it's the system who care about binding IPV4 too).
Now I have to find this service. Problem : I found it nowhere using netstat. Is there a way to force closing a socket?
[EDIT] I try stopping containers and start nginx directly on the server, bind on 443 +SSL IPV4/IPV6 and serve a "hello" file . => IPV6 works, IPV4 don't work => I change the port from 443 to 444 => IPV4 and IPV6 works => I clear IPTABLE and configure ACCEPT on everything => Still same problem with IPV4 443 :(
Because Nginx manage to bind on 0.0.0.0:443 I suppose the socket was "free". So something is blocking or filtering data incoming from IPV4 443
[edit] Finally it was an IPTABLE issue. I tag my answer as the solution to re-open port 443 in IPV4. I will look in my NAT table to find which rule is causing this and I will open a new topic if needed . tahnks everyone for yout ime and help !
I finally found a way to make it work. Just FLUSHED nat using the below command:
iptables -t nat -F
And now everything works. :)
I don't really understand what this command does. It flushes the NAT rules but I can't see any difference in my IPTABLE.
If someone can tell me if this "flush" is risky (I never changed something in IPTABLE and only docker change it).