how to prevent 502 status code as response by haproxy as load balancer

Question

I have 3 server:

server (A)= a nginx(port 80) as reverse proxy to kestler (5000 port)
server (B)= a nginx(port 80) as reverse proxy to kestler (5000 port)
server (C)= a HAProxy as load balancer for port 80 of server (A) and (B)
and server A & B are quite similar.

every things works very well and haproxy forwards requests to server (A) & (B), but if kestrel in one of servers (e.g. A) be killed, nginx respond 502 bad gateway error and haproxy not detect this issue and still redirect requests to it, and this is mistake! it must redirect requests to server (B) in this time.

global
    log 127.0.0.1 local2 info
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option redispatch
    retries 3
    timeout connect 5s
    timeout client 50s
    timeout server 50s
    stats enable
    stats hide-version
    stats auth admin:admin
    stats refresh 10s
    stats uri /stat?stats

frontend http_front
    bind *:80
    mode http
    option httpclose
    option forwardfor
    reqadd X-Forwarded-Proto:\ http
    default_backend http_back

backend http_back
    balance roundrobin
    mode http
    cookie SERVERID insert indirect nocache
    server ServerA 192.168.1.2:80 check cookie ServerA
    server ServerB 192.168.1.3:80 check cookie ServerB

How Can I resolve this issue? thanks very much.

Answer 1

You are only checking whether nginx is running, not whether the application is healthy enough to use.

In the backend, add option httpchk.

option httpchk GET /some/path HTTP/1.1\r\nHost:\ example.com

Replace some path with a path that will prove whether the application is usable on that server if it returns 200 OK (or any 2xx or 3xx response), and replace example.com with the HTTP Host header the application expects.

option httpchk

By default, server health checks only consist in trying to establish a TCP connection. When option httpchk is specified, a complete HTTP request is sent once the TCP connection is established, and responses 2xx and 3xx are considered valid, while all other ones indicate a server failure, including the lack of any response.

This will mark the server as unhealthy if the app is not healthy, so HAProxy will stop sending traffic to it. You will want to configure a check interval for each server using inter and downinter and fastinter options on each server entey to specify how often HAProxy should perform the check.