Search code examples
amazon-ec2vpnamazon-eksconnection-reset

ERR_CONNECTION_RESET from EKS nodes

I had EC2 server where I was running my existing application. The EC2 instance was on private subnet and ELB was created in public subnet with access to particular VPN IP. So whenever I was on VPN, I was able to access my application and if I am outside that VPN IP then I wasn't able to access the application.

Now I have created EKS cluster and I am deploying my application using kubectl with docker image of the application. Weird thing is the application works fine whenever I am NOT on VPN (I tweaked security group to allow all traffic from all IPs) and whenever I am on VPN, I receive "ERR_CONNECTION_RESET" in chrome and curl shows - empty response received from server.

Till now I have tried below things. As I am relatively new with EKS, I am not able to find much. 1. Same security group applied - Not resolving 2. Checked logs of all pods - whichever pods I received from "kubectl get po --all-namespaces" - No issues showing up 3. Checked /var/log/messages 4. Tried to change application port 5. kubectl get events not showing anything on why server is sending back empty response. 6. Tried to SSH to node and tried to curl localhost:30080 and it works fine, but when tried to curl from my machine (which is on VPN), it fails with empty response.

Please again note that, the application runs totally fine when I am outside VPN. Further my old application (that is on EC2) runs fine with VPN.

Thanks in advance!

Solution

  • Finally found the issue was with the corporate VPN which was blocking all ports other than 80 and 443. When I was creating the service, I wanted to have ELB to expose port 5000. So I was thinking elb-host:5000 will point to dev service nodeport which was 30080. This was perfectly working when I was NOT on the VPN. But when I was connecting the site using VPN, corporate traffic was blocking port 5000 on ELB. After I changed the port to 80, it started working as expected.

    While using nginx, it was creating ELB with port 80 instead of my intended port 5000. I didn't notice that port change and thought that this is happening due to IP blocking.