tcp-check expect binary response in second packet in a row

Question

I am trying to build a TCP checking on my backend servers using HAProxy version 1.5.8.

The behavior should be as follows:

1. Send binary data to server
2. Receive ACK as first packet
3. Receive confirmation data in second packet

So I need to check that after sending binary data I received ACK and after that other binary data in a second packet in a row.

Is it possible to do it with HAProxy.

I am trying to find it in documentation and also trying to create different configurations, unsuccessfully:

option tcp-check
tcp-check connect
tcp-check send-binary 303030303030
tcp-check expect binary 303030303030

Every time I received back from server ACK, connection is terminated by HAProxy with the result that the backend server is down.

EDIT:

I will receive the following:

First packet after sending data

0000   a0 66 10 09 2e 46 9c af ca bb aa 47 08 00 45 00    f...F.¯Ê»ªG..E.
0010   00 28 40 58 40 00 3e 06 d7 04 0a 1e 0b 34 0a 02   .(@X@.>.×....4..
0020   06 20 25 1c d5 80 91 0a f8 87 db 03 25 8f 50 10   . %.Õ...ø.Û.%.P.
0030   01 c9 03 d6 00 00 00 00 00 00 00 00               .É.Ö........

Second packet right after the above

0000   a0 66 10 09 2e 46 9c af ca bb aa 47 08 00 45 00    f...F.¯Ê»ªG..E.
0010   00 39 40 59 40 00 3e 06 d6 f2 0a 1e 0b 34 0a 02   .9@Y@.>.Öò...4..
0020   06 20 25 1c d5 80 91 0a f8 87 db 03 25 8f 50 18   . %.Õ...ø.Û.%.P.
0030   01 c9 2d 2e 00 00 00 0f 30 30 30 30 30 30 42 33   .É-.....000000B3
0040   30 30 43 48 45 43 4b                              00CHECK

The first is without any data and I need to check that the second contains 000000.

EDIT2:

PCAP provided:

Normal behavior when communication goes directly from client to server, without HAProxy: Normal behavior - client to server

Using HAProxy as load balancer, connecting to the same server and checking with the same command, failing to check: failing check - HAProxy to server

backend configuration:

backend nodes
        mode tcp
        balance roundrobin
        default-server inter 10s fall 3 rise 2
        option tcp-check
        tcp-check connect
        tcp-check send-binary 303030303030423230303035434845434b
        tcp-check expect binary 000f30303030303042333030434845434b
        server server1 10.30.11.52:9500 check
        server server2 10.30.11.52:9501 check
        server server3 10.30.11.52:9502 check

Answer 1

2. Receive ACK as first packet

HA proxy does not work at the raw packet level but at the TCP level. At this level there is no such thing as an ACK as a single packet. There is not even the concept of a packet at this level. Instead there is only the concept of a data stream consisting of the received bytes.

Every time I received back from server ACK, connection is terminated by HAProxy with the result that the backend server is down.

Given that HA proxy does not care about packets with zero payload in the first place it is likely that your "ACK as first packet" is actually some packet which contains an ACK (as almost all TCP packets do) but also contains some payload, but not the one you expect with the "next packet". Since the payload does not match the payload you specify as expected the check fails.

Note that this is only an assumption made based on incomplete information about your "ACK as first packet". To prove the assumption one would actually need to see what is really going on on the wire, for example by having a packet capture.

EDIT#1:
after the OP provided a some (undocumented) dump of the packets and some figuring out where the actual IP header in these packets starts (offset 14, i.e. prefixed with layer 2 ethernet header) it is clear that the first packet has no payload which means it gets completely ignored by the check. The second packet then has the following payload of 17 bytes:

0030                     00 0f 30 30 30 30 30 30 42 33         ..000000B3
0040   30 30 43 48 45 43 4b                              00CHECK

Given that the OP checks for binary 303030303030 but the actual payload is 00 0f 30 30 30 30 30 30 .... the given tcp-check expect ... does not match the actual payload and thus the check fails.

EDIT#2:
After the OP has provided the pcap of a connection without and with haproxy a difference in the behavior of both client/haproxy and server can be seen:

1. without haproxy:

   • client sends 2 bytes \x00\x11 to the server followed by 17 bytes \x30\x30....
   • server replies immediately with 17 bytes \x00\x0f\x30\x30....

2. with haproxy:

   • haproxy send 17 bytes \x30\x30... to the server.
     It does not send the initial 2 bytes \x00\x11 as done by the original server !!!
   • Server does not reply (except an ACK with no payload). After 6 seconds of inactivity haproxy closes the connection to the server and likely considers the check failed.

In summary: I think the haproxy check fails to send the proper request to the server, i.e. the first 2 bytes are missing. That's why the server will not respond at all and the check will fail after some timeout.