Search code examples
amazon-web-servicesnginxamazon-ec2.net-corekestrel-http-server

Why do I get a 502 when I attempt to access ELB DNS


I am attempting to host an ASP.NET Core application on an EC2 instance with the following setup (from top to bottom). Given the setup below, I get a health status on the EC2 instance as well as the Target Group. Also, if I check the status of the Kestrel service on the server, I can see incoming requests from the Target Group's health checks with 200 responses.

The problem that I'm having is that if I access the ELB's DNS endpoint, I get a 502 (Bad Gateway) response. Why is this? Especially since I see on the web server the requests for the health checks coming in and met with 200 responses. I don't however see the requests that I make explicitly against the ELB's DNS endpoint. It's like the requests to the ELB's DNS endpoint never actually reach the web server.

Another thing that to take not of is that I have a jump server in a public subnet within the same VPC as the private subnet housing the EC2 instance. I am able to make a requests from the jump server to the private IP and get back a 200 response with my login html. It would seem that everything on the web server is correct; something is wrong with something outside of that.

Route 53

I'm using a Route 53 hosted zone for my domain. The HZ has an A record that points to a Elastic Load Balancer.

Elastic Load Balancer

Traffic from the domain (on ports HTTP and HTTPS) gets routed to an Application Load Balancer listening to HTTP (80) and HTTPS (443). The HTTPS traffic has a Certificate Manager-assigned public SSL cert. The HTTPS traffic terminates here and is passed through to underlying instances as HTTP.

ELB Target Group

I have a target group pointing a specified instance. For the target group, I'm targeting the login page of the running application and expecting a 200 response.

Auto-Scaling Group

I have an ASG that points to the ELB Target Group.

Web Server

For the web server, I'm running an Amazon Linux 2 AMI Instance with Kestrel running behind an Nginx reverse proxy. The Kestrel service hosts an ASP.NET MVC application running on port 5000 (HTTP).

The EC2 instance is sitting in a private subnet. There's a public subnet within the same VPC and AZ with an Internet Gateway attached.

For the .NET Core application's Startup Config, I am using standard forwarding headers.

There is a Security Group attached to the EC2 instance allowing HTTP and HTTPS traffic from the ELB security group and SSH traffic from a jump box sitting in a public subnet in the same VPC.


Solution

  • What I ended up finding was that my Nginx configuration was conflicting with with the ELB configuration. I ended up restoring the nginx.conf back to its initial state and provided more clearly defined custom configurations under the conf.d folder (I created a .conf file here specifically for my app that routes traffic the way that I want).

    I also did the following to streamline my intentions of each pieces' role:

    1. Since SSL terminates at the ALB, I modified Kestrel to no longer listen for 443 (HTTPS) traffic; it only listens for 80 (HTTP) traffic. For now, this is fine because there aren't any other members of the VPC which would pose a security threat due to the HTTP traffic past the ELB. Above the ELB, everything is still encrypted.

    2. I added an ELB listener that listens for both 443 and 80 traffic and forwards it to a target group which targets 80 only (remember, Kestrel is now only looking for 80 traffic).

    3. I also needed to change startup.csto not UseHttpRedirection; again, SSL terminates at the ELB.

    A combination of all of these things resolved my issue.