Search code examples
amazon-web-servicesamazon-ec2iptablesnat

ICMP request not been forwarded by NAT instance

I'm in the process of setting up a NAT-instance on AWS EC2. I've taking inspiration from fck-nat a project aiming to replace the default nat-gateway.

So far I've two working instances, each in a separated subnet. One instance is with a public IP and is connected to the internet gateway. The other is in a "private" subnet with routing to the first instance.

I can ping 1.1.1.1 (It's the address I use for testing) from the nat-instance. I can ping the nat-instance from the private subnet.

But the forwarding seem to no be working, I've followed the above script as well as the instruction (quite deprecated) from aws documentation nat instance.

I made sure that:

  1. The routing is working properly (it seems to be the case)
  2. I allowed ip forwarding using sysctl -p file_name and net.ipv4.ip_forward=1 in the said file.
  3. Using iptables -t nat -A POSTROUTING -o ens5 -j MASQUERADE
  4. Disabling source/destination on the nat-instance.

My suspect would be Security Group, but pinging from the nat-instance is working and applying the same security to the private instance doesn't solve the problem.

I'm suspecting an underlying security I'm not aware of, or maybe my nat setup isn't complete.

Following some logs I took, showing what I get from tcpdump while been on the nat-instance.

  • XX.XX.AA.AA would be the ip of the nat-instance.
  • XX.XX.AA would be the network of the nat-instance subnet.
  • XX.XX.BB.BB would be the ip of the instance in the private subnet.
  • XX.XX.BB would be the network of the private subet.
[user_name@ip-XX-XX-AA-AA ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 06:1e:58:67:bb:a9 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-033f1bdf14d0a2f0a
    altname device-number-0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:f5:9d:a2:cf brd ff:ff:ff:ff:ff:ff

[user_name@ip-XX-XX-AA-AA ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       tcp  --  anywhere             anywhere             tcp dpt:51678
DROP       all  -- !127.0.0.0/8          127.0.0.0/8          ! ctstate RELATED,ESTABLISHED,DNAT
DROP       tcp  --  anywhere             anywhere             tcp dpt:51678
DROP       all  -- !127.0.0.0/8          127.0.0.0/8          ! ctstate RELATED,ESTABLISHED,DNAT

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

[user_name@ip-XX-XX-AA-AA ~]# tcpdump -n net XX.XX.BB
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:50:12.566281 IP XX.XX.BB.BB > 1.1.1.1: ICMP echo request, id 7, seq 4, length 64
09:50:13.606290 IP XX.XX.BB.BB > 1.1.1.1: ICMP echo request, id 7, seq 5, length 64
09:50:14.646280 IP XX.XX.BB.BB > 1.1.1.1: ICMP echo request, id 7, seq 6, length 64

--- EDIT --- Adding the required output from tracepath as well as the route table of the instance residing in the private subnet.

[user_name@ip-XX-XX-BB-BB ~]# tracepath 1.1.1.1
 1?: [LOCALHOST]                      pmtu 9001
 1:  XX.XX.AA.AA                                           0.986ms asymm  2 
 1:  XX.XX.AA.AA                                           0.883ms asymm  2 
 2:  no reply
 3:  no reply

[user_name@ip-XX-XX-BB-BB ~]# ip route
default via XX.BB.BB.1 dev ens5 proto dhcp src XX.XX.BB.BB metric 512 
XX.XX.0.2 via XX.XX.BB.1 dev ens5 proto dhcp src XX.XX.BB.BB metric 512 
XX.XX.BB.0/24 dev ens5 proto kernel scope link src XX.XX.BB.BB metric 512 
XX.XX.BB.1 dev ens5 proto dhcp scope link src XX.XX.BB.BB metric 512 
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

If the route table seems uncorrect, I use the AWS route table and designate the nat-instance as the gateway for 0.0.0.0/0 as such: enter image description here

Solution

  • After investigating and reading some post I found out I forgot to allow the packets to pass :

    • iptables -A FORWARD -i ens5 -o ens5 -m state --state RELATED,ESTABLISHED -j ACCEPT
    • iptables -A FORWARD -i ens5 -o ens5 -j ACCEPT

    After adding executing those commands the ping response is received by the instance within the private subnet, and a quick check using ipconfig.io and tracepath indicate it is using the public ip from the nat-instance.