I'm running into an issue with latency when using a set of micro-apps on EB with the load balancer. This issue is specifically with one nginx powered app trying to hit another app on ELB. If we hit the API using postman it's fast, but posting to the endpoint through nginx-lua is incredibly slow comparatively.
My requests are being proxy_passed by nginx-lua (openresty) for security purposes, along with a rewrite being done in nginx. Ex (changed paths and variable names)
location /api
{
proxy_set_header Authorization $token;
rewrite /api/(.*) /ipa/$1 break;
proxy_pass $elb_api_cname;
}
It seems that the first few requests to a specific api (we have several) all seem to have a lot of latency which will then go down briefly for subsequent requests.
These dev instances don't get a lot of traffic. When we change the proxy_pass variable being set by lua to the direct IP of an EC2 instance it's almost 4-10x faster (300ms consistent vs 1-5s).
I believe it has something to do with nginx caching the IP on start and the ELB changing IP's when it gets a little bit of traffic vs no traffic)
So the issue ended up being a problem we could fix by updating our resolvers.
In the location block we specified the resolvers for amazons DNS service, while we left the other resolvers the same to work for some internal proxy stuff.
This is now giving us the expected speed.