I'm experiencing some rather peculiar behaviour on a machine which has both Jenkins and Docker installed. For clarity, Jenkins is not running as a Docker container but runs under the jenkins user.
When running curl in a container, I get a 403:
root@ada71c8116bf:/# curl -I www.google.co.uk
HTTP/1.1 403 Forbidden
Date: Tue, 30 May 2017 13:41:07 GMT
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.f1223778=36hjq9sozhveoe1bfsss1dnq;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/html;charset=UTF-8
X-Hudson: 1.395
X-Jenkins: 2.46.3
X-Jenkins-Session: 2836b130
X-You-Are-Authenticated-As: anonymous
X-You-Are-In-Group-Disabled: JENKINS-39402: use -Dhudson.security.AccessDeniedException2.REPORT_GROUP_HEADERS=true or use /whoAmI to diagnose
X-Required-Permission: hudson.model.Hudson.Read
X-Permission-Implied-By: hudson.security.Permission.GenericRead
X-Permission-Implied-By: hudson.model.Hudson.Administer
Content-Length: 793
Server: Jetty(9.2.z-SNAPSHOT)
Outside the container on the host, I get the expected response:
$ curl -I www.google.co.uk
HTTP/1.1 200 OK
Date: Tue, 30 May 2017 13:40:17 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=104=mMKjBy002X3N_SkhkD_8xuAwpFuw03CFi0iOJjNX81FUHfMT6qTq95LcgRwdhrV_GZoUF9LQ1B9qAQPriN9Er3Bu2JWoqPgvt16TduuVj5QsNs9GiJTQBtaSXWic7G9E; expires=Wed, 29-Nov-2017 13:40:17 GMT; path=/; domain=.google.co.uk; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
Jenkins is obviously to blame but I've got no idea why it would be intercepting HTTP traffic leaving containers. Pinging Google works fine, so does sending HTTPS requests. No other machine possesses this issue (presumably because they don't have Jenkins installed). So, what's going on here? How do I get Jenkins to stop intercepting HTTP from Docker containers?
Update
Turning off Jenkins' "Prevent Cross Site Request Forgery exploits" option causes Jenkins to no longer return 403s. Instead, Jenkins responds to any HTTP request from within a container with the dashboard page, i.e. the default page.
Also worth noting is that DNS works fine; hostnames are resolved to the correct IP addresses.
I'm going to get out Wireshark.
Through using Wireshark I found that something was redirecting HTTP traffic to port 8090 on the host. A lucky google led me to check the host's IP tables (iptables -t nat -L -n) and sure enough there were rules that redirected all port 80 traffic from anywhere to port 8090 of the host. Someone had clearly set up this redirect for the benefit of Jenkins users.
The solution was to alter the IP tables to not redirect traffic coming from the docker subnet.
The tables before:
$ sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 redir ports 8090
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- 0.0.0.0/0 127.0.0.1 tcp dpt:80 redir ports 8090
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Commands to alter:
$ sudo iptables -t nat -R PREROUTING 1 ! -s 172.17.0.0/16 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8090
$ sudo iptables -t nat -R OUTPUT 1 ! -s 172.17.0.0/16 -d 127.0.0.1/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8090
Resulting IP tables:
$ sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- !172.17.0.0/16 0.0.0.0/0 tcp dpt:80 redir ports 8090
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- !172.17.0.0/16 127.0.0.1 tcp dpt:80 redir ports 8090
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0