Nginx Load Balancer issue

Question

We are using Nginx As a load balancer for multiple riak nodes. The setup worked fine for some time(few hours) before Nginx started giving bad gateway 502 errors. On checking the individual nodes seemed to be working. We found out that The problem was with nginx buffer size hence increased the buffer size to 16k, it worked fine for one more day before we started getting 502 error for everything.

My Nginx configuration is as follows

upstream riak {
   server 127.0.0.1:8091 weight=3;
   server 127.0.0.1:8092;
   server 127.0.0.1:8093;    
   server 127.0.0.1:8094;

  }

  server {
    listen 8098;
    server_name 127.0.0.1:8098;
    location / {
      proxy_pass http://riak;
      proxy_buffer_size 16k;
      proxy_buffers 8 16k;
    }
  }

Any help is appreciated,Thank you.

Answer 1

Check if you are running out of fd's on the nginx box. Check with netstat if you have too many connections in the TIME_WAIT state. If so, you will need to reduce you tcp_fin_timeout value from default 60 seconds to something smaller.