I have a network application built using Netty. The application is behind Amazon network load balancer.
I now want to be able to retrieve the original client IP address, so I turned on the Proxy Protocol v2 setting on the network load balancer.
Unfortunately doing this breaks the application.
The application can be interacted with, via the terminal using something like nc or telnet.
On connecting to the application, the user will normally receive a welcome message and then a prompt to type in the query to interact with the application.
When the Proxy Protocol v2 is turned on, on connection, the welcome message is no longer written out to the client, instead the client sees something like this:
telnet -4 app.host 43
Trying <IP ADDRESS>...
Connected to some_network_lb.elb.eu-west-1.amazonaws.com.
Escape character is '^]'.
capturing the network packet I noticed a difference between when the Proxy protocol v2 is switched off (Application works fine) and when it is switched on (Application does not work fine)
When the Proxy protocol v2 is switched off, the packet interaction can be summarized as follows:
No. Time Source Destination Protocol Length Info port
1 0.000000 SOURCE-IP DEST-IP TCP 78 57059 → 43 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1158853985 TSecr=0 SACK_PERM=1 43
No. Time Source Destination Protocol Length Info port
2 0.034526 DEST-IP SOURCE-IP TCP 74 43 → 57059 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=1420 SACK_PERM=1 TSval=3185991482 TSecr=1158853985 WS=128 57059
No. Time Source Destination Protocol Length Info port
3 0.034556 SOURCE-IP DEST-IP TCP 66 57059 → 43 [ACK] Seq=1 Ack=1 Win=132352 Len=0 TSval=1158854019 TSecr=3185991482 43
No. Time Source Destination Protocol Length Info port
4 0.067278 DEST-IP SOURCE-IP TCP 264 43 → 57059 [PSH, ACK] Seq=1 Ack=1 Win=26880 Len=198 TSval=3185991515 TSecr=1158854019 [TCP segment of a reassembled PDU] 57059
No. Time Source Destination Protocol Length Info port
5 0.067301 SOURCE-IP DEST-IP TCP 66 57059 → 43 [ACK] Seq=1 Ack=199 Win=132096 Len=0 TSval=1158854051 TSecr=3185991515 43
And when the Proxy protocol v2 is switched on, the packet interaction can be summarized as follows:
No. Time Source Destination Protocol Length Info port
1 0.000000 SOURCE-IP DEST-IP TCP 78 55871 → 43 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1158086997 TSecr=0 SACK_PERM=1 43
No. Time Source Destination Protocol Length Info port
2 0.040204 DEST-IP SOURCE-IP TCP 74 43 → 55871 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=1420 SACK_PERM=1 TSval=3185217396 TSecr=1158086997 WS=128 55871
No. Time Source Destination Protocol Length Info port
3 0.040227 SOURCE-IP DEST-IP TCP 66 55871 → 43 [ACK] Seq=1 Ack=1 Win=132352 Len=0 TSval=1158087035 TSecr=3185217396 43
As can be seen above, when the Proxy Protocol v2 is turned on, the packet exchange stops at the third exchange, and the server never sends the 4th package exchange, which contains the [PSH, ACK] when the Proxy Protocol v2 is turned off.
Any idea why this is happening? Why is it that, the packet with the [PSH, ACK] flag is never sent when Proxy Protocol v2 is turned on? And any tips on how to fix this?
We had to contact AWS support to help with setting the target group attribute
proxy_protocol_v2.client_to_server.header_placement to on_first_ack on the NLB target group.
By default trying to set the attribute does not work. But once AWS has enabled it, the command to update the attribute will look like this:
“aws elbv2 modify-target-group-attributes --target-group-arn arn:aws:elasticloadbalancing:us-east-2*************************** --attributes ‘{“Value”: “on_first_ack”, “Key”: “proxy_protocol_v2.client_to_server.header_placement”}’ ”
For us, this was due to the nature of our application, where no data is sent from the client after the connection is made, and this does not work out of the box because headers are pushed lazily.
You can read this link for more info on this behaviour