nginx as reverse proxy returns 502s when upstream server is behind an ALB

Question

My setup is a load balancer going to an nginx box which acts as a reverse proxy to a load balancer that sits in front of a uwsgi application.

I get a sporadic 502s and quite a few

epoll_wait() reported that client prematurely closed connection

Looking at the ALB logs for the one in front of the uwsgi application, the response_processing_time field is set to -1. I have tested that bypassing nginx and going straight to the ALB to talk to the uwsgi application runs fine, so something from the nginx side is breaking.

I looked at dozens of solutions and here are the main ones I tried without avail

• Tried turning off keepalive
• Tried increasing keepalive timeout
• Made sure it's on http1.1
• Increased proxy read/connection timeout
• Upgraded to Nginx 1.17.3
• Increased ulimit, somaxconn, etc

• Set a lot of headers and settings in the location

  location @uwsgi {
   proxy_http_version 1.1;
   proxy_set_header Connection "";
   proxy_cache_key sfs$request_uri$scheme;
   proxy_pass $uwsgi_django_backend;
   proxy_set_header X-NginX-Proxy true;
   proxy_set_header Connection "";
   proxy_set_header X-Request-Start "t=${msec}000";
   proxy_set_header Host $host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Proto $scheme;
  }
  

I tried removing the uwsgi ALB and going straight to a single UWSGI application. 502's went away. Obviously this is not going to work as I scale.

Does anyone have any ideas why the ALB is giving me a bad response?

Answer 1

I managed to fix this problem by forcing the upstream service to use HTTP1.1

In my context, I changed the environment variable HTTP11_SOCKET to UWSGI_HTTP11_SOCKET.