I have an application running in an EC2 instance in a private subnet (to add extra security), receiving traffic directly from an internet-facing NLB that is associated to the public subnet.
I have also configured a NAT Gateway in the public subnet so that the private EC2 instance can download whatever needed from the Internet.
I have just come out to the next conclusion:
If I request from the Internet: http://index.html
The private EC2 Instance uses the NAT Gateway to send back the HTTP response, and therefore you are charged for that NAT processing.
https://aws.amazon.com/vpc/pricing/
"Data processing charges apply for each Gigabyte processed through the NAT gateway regardless of the traffic’s source or destination"
The Route Table associated to the Private Subnet (where the web/app server is located) has { - local ; 0.0.0.0/0 - NATGateway} If I remove the 0.0.0.0/0 entry, then the HTTP requests to the server do not work. And if I remove the NAT Gateway I get the same issue. Additionally, when I have the NAT Gateway and the route table to use it from the Private Subnet, I can also see traffic in the monitoring tab of the NAT Gateway when I do a simple HTTP request from the Internet - http:///index.html
Has someone faced the same issue? Is my understanding correct?
Is there any workaround to avoid this? I can just think of the following:
Move the application to a Web Tier (in a public subnet) so that the EC2 Instance has a public IP and therefore it does not need the NAT Gateway to respond to every HTTP request
Create a Web Tier in addition to the Application Tier, so that all traffic goes from the NLB <-> Web Tier <-> App Tier
Create a NAT Instance instead of a NAT Gateway so that you are not charged for that NAT processing.
Thanks!
So, it turns out there is an strange behaviour (bug?) in NLB that requires a default route to something for return traffic to work.
Scenario:
Sending traffic to the NLB will not give a response.
Add:
This works.
However, it doesn't actually use the NAT Gateway!
Instead of adding a NAT Gateway, you can instead create a default route to another instance, for example:
0.0.0.0/0 -> another instance
Then, connecting to the NLB works!
Behind the scenes, the VPC network detects that traffic coming out of the instance is in response to a request that came via the NLB, and traffic will be routed out the NLB.
HOWEVER, it only works if there is a valid default route in the route table. The route isn't used -- the network overrides it, so it doesn't matter what it points to. However, it must point to a valid resource. It doesn't work if it points to an ENI that isn't attached to anything. In my tests, it also doesn't seem to work if you point it back to the same instance. But, I pointed it to another instance and it worked just fine.
AWS Support agrees that this is not the best behaviour, but it's how things have to be configured for the moment.
Bottom line: It does not require a NAT Gateway. It just needs a valid default route that will be ignored when routing return traffic.