Search code examples
apachehaproxy

haproxy 504 timeout to apache


Very new to haproxy and loving it, apart from a 504 issue that we're getting. The relevant log output is:

Jun 21 13:52:06 localhost haproxy[1431]: 192.168.0.2:51435 [21/Jun/2017:13:50:26.740] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 13:54:26 localhost haproxy[1431]: 192.168.0.2:51447 [21/Jun/2017:13:52:46.577] www-https~ beFootprints/foorprints 0/0/3/-1/100005 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:15:57 localhost haproxy[1431]: 192.168.0.1:50225 [21/Jun/2017:14:14:17.771] www-https~ beFootprints/foorprints 0/0/2/-1/100004 504 195 - - sH-- 3/3/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:22:26 localhost haproxy[1431]: 192.168.0.1:50258 [21/Jun/2017:14:20:46.608] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1" 

Using the following timeout values in the haproxy.cfg

defaults
        log     global
        mode    http
        option forwardfor
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  100000

Running on Ubuntu 16.04.2 LTS

Any help and comment very much appreciated!


Solution

  • The problem appears to be with the web server. Check the logs, there, and you should find long-running requests.

    Here's how I conclude that.

    Note sH-- in your logs. This is the session state at disconnection. It's extremely valuable for troubleshooting. The values are positional and case-sensitive.

    s: the server-side timeout expired while waiting for the server to send or receive data.

    ...so, timeout server fired, while...

    H: the proxy was waiting for complete, valid response HEADERS from the server (HTTP only).

    The server had not finished (perhaps not even started) returing all the response headers to the proxy, but the connection was established and the request had been sent.

    HAProxy returns 504 Gateway Timeout, indicating that the backend did not respond in a timely fashion.

    If your backend needs longer than 100 seconds (?!) then you need to increase timeout server. Otherwise, your Apache server seems to have a problem being too slow to respond.