Search code examples
apachemod-proxyx-forwarded-for

What determines when mod_proxy only sometimes add internal ip addresses to X-Forwarded-For header

I've been experimenting with Apache mod_proxy and mod_remoteip in order to confirm my understanding of the handling of X-Forwarded-For headers, particularly around how internal IP Addresses (e.g. 10.x.x.x or 192.168.x.x ranges) are being handled.

It seems like mod_proxy doesn't always add the internal IP addresses to the X-Forwarded-For header, but I've been unable to find any documentation explaining the expected behavior for this.

As far as I can tell, when the request initiates from an internal IP address, then mod_proxy will add internal IP addresses to the X-Forwarded-For header, but when the initial request comes from a public IP, mod_proxy does not seem to add any internal IP addresses to the X-Forwarded-For.

The Question

My question is: What are the rules that govern whether or not mod_proxy will append the calling IP address to the X-Forwarded-For header.

The documentation on mod_proxy says:

When acting in a reverse-proxy mode (using the ProxyPass directive, for example), mod_proxy_http adds several request headers in order to pass information to the origin server. These headers are:

X-Forwarded-For - The IP address of the client.

X-Forwarded-Host - The original host requested by the client in the Host HTTP request header.

X-Forwarded-Server - The hostname of the proxy server.

Be careful when using these headers on the origin server, since they will contain more than one (comma-separated) value if the original request already contained one of these headers. For example, you can use %{X-Forwarded-For}i in the log format string of the origin server to log the original clients IP address, but you may get more than one address if the request passes through several proxies.

I read this as saying that the client IP address will always be appended to the X-Forwarded-For header, but that's not the behavior I'm observing.

The rest of this question is the tests I've conducted and the behavior I've observed.

The Setup

I've setup two servers both running Apache with mod_proxy installed. I'll refer to these as One and Two.

  • One has the (internal) IP address 10.0.7.1
  • Two has the (internal) IP address 10.0.7.2

One has the following ProxyPass directive so that requests to sub-paths of /proxyToTwo are sent to the equivalent sub-path under /proxyToOne on Two

<Location "/proxyToTwo">
        ProxyPass http://10.0.7.2/proxyToOne
</Location>

Two has the following ProxyPass directive so that requests to sub-paths of /proxyToOne are sent back to One but without the /proxyToOne prefix

<Location "/proxyToOne">
        ProxyPass http://10.0.7.1
</Location>

The effect of this is that when I issue requests to http://One/proxyToTwo/foo it's proxied as follows

  1. One receives the request, issues the following request to Two http://10.0.7.2/proxyToOne/foo
  2. Two receives the request, issues the following request back to One http://10.0.7.1/foo
  3. One receives a request for /foo and actually serves the resource

So every request is bounced from one to two and back to one before being responded two.

Calling with an Internal IP

Using the above setup, I call One from Two using it's internal IP address:

curl http://10.0.7.1/proxyToTwo/foo

The X-Forwarded-For and X-Forwarded-Host headers received when One eventually gets the request for the /foo resource is what I expect below:

X-Forwarded-For: 10.0.7.2, 10.0.7.1
X-Forwarded-Host: 10.0.7.1, 10.0.7.2

This is what I expect, that the request was proxied first through One then Two, and the requesting IP addresses are first the initial request from Two (curl) then the request from One (mod_proxy) and the final request (not in the header because it's the client IP of the connection being from Two (mod_proxy)

Calling with external IP

The unexpected behavior is that mod_proxy seems to behave differently when called from public IP. So Instead of calling One from Two, I call One from my local machine using the public address

curl http://35.162.28.102/proxyToTwo/foo

The X-Forwarded-Host is still what I expect:

X-Forwarded-Host: 35.162.28.102, 10.0.7.2

That is, the request was first proxied through One (using it's external address) then through Two.

But the X-Forwarded-For header is showing only my (external) IP address:

X-Forwarded-For: 35.163.25.76

This suggests to me that the initial execution of mod_proxy is adding the X-Forwarded-For header with the client IP address. But then the subsequent proxying by Two doesn't append the address of One.

I think this behavior is probably more useful than blindly appending the internal IP addresses to the header, but I can't find it documented anywhere so would like to ensure I fully understand it.

Solution

  • Answering this so it's available in-case others make a similar mistake.

    In short: mod_proxy always appends the client IP to the X-Forwarded-For header (or adds an X-Forwarded-For header if there is no existing header).

    But, other Apache modules can manipulate the X-Forwarded-For header prior to mod_proxy processing it. Mod_proxy will append the client IP to what it sees as the X-Forwarded-For header, which can be the output of other Apache modules.

    In my tests, the other module affecting the result was mod_remoteip. The reason for the difference in resulting behavior was that I was using the RemoteIpTrustedProxy directive to specify my trusted proxies and this will allow trusting of private IPs as the first connection but will not process private IPs in the X-Forwarded-For header.

    As a result for the External IP case there was the following handling:

    1. My Machine (35.163.25.76) connects to One with no X-Forwarded-For
    2. One sends this request to Two with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)
    3. Two receives this request, mod_remoteip processes the X-Forwarded-For header, because One is trusted, the client IP effectively is 35.163.25.76 and there is effectively no X-Forward-For header passed on to mod_proxy
    4. Two sends this request to One with my IP in X-Forwarded-For header (X-Forwarded-For: 35.163.25.76)

    This is what presents as "not appending the private IP" but is actually processing the X-Forward-For header and producing an identical one.

    The Private ID case behaves differently because mod_remoteip isn't accepting the private IP address in the X-Forward for. So it's handled by:

    1. Two (10.0.7.2) connects to One with no X-Forwarded-For
    2. One sends this request to Two with Two's IP in X-Forwarded-For header (X-Forwarded-For: 10.0.7.2)
    3. Two receives this request, mod_remoteip processes the X-Forwarded-For header but does not trust the value so leaves it as is. So the client IP remains One's IP (10.0.7.1) and the X-Forward-For header passed on to mod_proxy is unmodified
    4. Two sends this request to One appending the client IP (One's) to X-Forwarded-For header resulting in X-Forwarded-For: 10.0.7.2, 10.0.7.1

    I verified this by actually sending in some X-Forwarded-For headers from trusted hosts. E.g. 

    curl http://10.0.7.1/proxyToTwo/foo --header "X-Forwarded-For: TrustedHost1, TrustedHost2"

    This results in the final X-Forward-For header containing only TrustedHost1 indicating that the header really is being processed by mod_remoteip and re-issued by mod_proxy.