Search code examples
tcpcentostimeoutamqphaproxy

HAProxy closes long living TCP connections ignoring TCP keepalive


I have configured HAProxy (1.5.4, but I tried also 1.5.14) to balance in TCP mode two server exposing AMQP protocol (WSO2 Message Broker) on 5672 port. The clients create and use permanent connection to the AMQP Servers, via HAProxy.

I've changed the client and server TCP keepalive timeout, setting net.ipv4.tcp_keepalive_time=120 (CentOS 7).

In HAProxy I've setted timeout client/server to 200 seconds (>120 seconds of the keepalive packets) and used the option clitcpka.

Then I've started wireshark and sniffed all the tcp traffic: after the last request from the clients, the tcp keepalived packets are sente regularly after 120 seconds, but after 200 seconds after the last request from the clients the connection are closed (thus ignoring the keepalived packet).

Below the configuration:

haproxy.conf

global
    log 127.0.0.1   local3
    maxconn 4096
    user haproxy
    group haproxy
    daemon
    debug

listen messagebroker_balancer 172.19.19.91:5672
    mode tcp
    log global
    retries 3
    timeout connect 5000ms
    option redispatch
    timeout client 200000ms
    timeout server 200000ms
    option tcplog
    option clitcpka 
    balance leastconn
    server s1 172.19.19.79:5672 check inter 5s rise 2 fall 3
    server s2 172.19.19.80:5672 check inter 5s rise 2 fall 3

Solution

  • TCP keep alive is at the transport layer and is only used to do some traffic on the connection so intermediate systems like packet filters don't loose any states and that the end systems can notice if the connection to the other side broke (maybe because something crashed or a network cable broke).

    TCP keep alive has nothing to do with the application level idle timeout which you have set explicitly to 200s:

    timeout client 200000ms
    timeout server 200000ms
    

    This timeouts gets triggered if the connection is idle, that is if no data get transferred. TCP keep alive does not transport any data, the payload of these packets is empty.