Search code examples
load-balancinghaproxy

HAproxy passive health checking

I'm new to haproxy and load balancing. I want to see what happens when a backend host is turned off while the proxy is running.

The problem is, if I turn off one of the backends and refresh the browser the page immediateltly exposes a 503 error to the user. After the next page load, it no longer gets the error since presumably that backend has been removed from the pool.

As a test I have set up two backend Flask apps and configured HAProxy to balance them like so:

backend app
    mode    http
    balanace    roundrobin
    server app1 127.0.0.1:5001 check
    server app2 127.0.0.1:5002 check

My understanding according to this: https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html#check-parameters

is that every 2 seconds a the backend hosts are pingged to see if they are up. Then they are removed from the pool if they are down. The 5xx error happens between the time I kill the backend and the 2 seconds.

I would think there is a way to get around this 5xx error by having HAProxy perform a little logic such that if a request from the frontend fails, it would then remove that failed backend from the pool and then switch to another and make another request. This way the user would never see the failure.

Is there a way to do this, or should I try something else so that my user does not get an error?

Solution

  • By default haproxy will retry 3 times (retries) with 1s intervals to the same backend. In order to allow to take another backend you should set option redispatch.

    Also consider to (carefully, it can be hamrful):

    Note: Haproxy retries only on connection errors (e.g. ECONNNREFUSED like in your case), it will not resend/resubmit request/data.