Disclaimer:
I am a networking newbie. Forgive me if I don't use proper vocabulary to describe my problem.
Context:
So I have a couple of app containers (Docker) that are to run on CoreOS. Each container must have different IAM roles. Since there will be potentially be N containers with N different IAM roles running on the same host, this poses a problem to use the EC2 Instance Metadata (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) to retrieve the instance credentials. Each container needs a different set of credentials (since each container will have different roles).
In order to fix that problem, I stumbled upon this project https://github.com/dump247/docker-ec2-metadata which is a Go proxy that intercepts (using iptables rules) calls to http://169.254.169.254/latest/meta-data/iam/security-credentials/ and provides proper credentials based on the calling container’s IAM role.
The iptables rule (see https://github.com/dump247/docker-ec2-metadata/blob/master/README.md) to reroute calls is:
# Get the host IP address. You can use a different mechanism if you wish.
# Note that IP can not be 127.0.0.1 because DNAT for loopback is not possible.
PROXY_IP=$(ifconfig eth0 | grep -Eo "inet addr:[0-9.]+" | grep -Eo "[0-9.]+")
# Port that the proxy service runs on. Default is 18000.
PROXY_PORT=18000
# Drop any traffic to the proxy service that is NOT coming from docker containers
iptables \
-I INPUT \
-p tcp \
--dport ${PROXY_PORT} \
! -i docker0 \
-j DROP
# Redirect any requests from docker containers to the proxy service
iptables \
-t nat \
-I PREROUTING \
-p tcp \
-d 169.254.169.254 --dport 80 \
-j DNAT \
--to-destination ${PROXY_IP}:${PROXY_PORT} \
-i docker0
Since we’re running on CoreOS, we want to deploy that proxy inside a Docker container. We don’t want to install the proxy directly on the EC2 host.
Suppose I have 2 containers running 2 apps with different IAM roles:
When the proxy intercepts a request to http://169.254.169.254/latest/meta-data/iam/security-credentials/ , the proxy needs to know the requesting container’s IP (e.g. 172.17.0.8), in order to use the Docker client (https://github.com/fsouza/go-dockerclient) to retrieve the requesting container’s IAM role (which is passed to the container using an environment variable, e.g.: docker run -e IAM_ROLE=arn:aws:iam::123456789012:role/roleA --name containerA myapp)
Problem:
Retrieving the requesting container’s IP (in our example, 172.17.0.[7, 8]) works perfectly fine when the proxy runs natively on the EC2 host. Here's my mental model of what happens when the proxy runs natively on the host:
containerA calls http://169.254.169.254 -> iptables routing happens -> call is redirected to localhost:18000 -> the proxy container sees containerA's IP as the incoming request's source
However, it does not work so well when the proxy runs inside a Docker container. Note that I start the proxy using docker run -p 18000:18000 -v /var/run/docker.sock:/var/run/docker.sock ec2-metadata-proxy
When the proxy runs in a Docker container, the requesting container’s IP is that of the docker0 interface (e.g. 172.17.42.1), making it impossible to determine the requesting container. Again, here's my mental model of what happens when the proxy runs in a container:
containerA calls http://169.254.169.254 -> iptables routing happens -> call is redirected to localhost:18000 -> Docker routes that call to the proxy container -> the proxy container sees docker0's IP as the incoming request's source
Question:
How can I make sure that the IP address that the proxy will see is that of the requesting container (e.g. containerA), and not that of the docker0 interface?
After a bit of experimenting, I found that running the proxy container in --net=host mode solved the problem.
By using this option, the proxy container shares the same networking stack as the host, thus bypassing the the docker0 bridge.