Search code examples
c++linuxsocketstcplisten

TCP server not accepting right number of connections issued by client with small listen backlog

I wrote a simple client program to issue a total of 10000 connections using 100 threads in a very small duration. And a simple server program with listen backlog set to 20 using epoll just to accept any new connection and record total connections number. I used ulimit -n to make sure it is larger than 20000 so there should be enough fd resource.

But after the server program accepted like 8800, 9400 (not fixed) connections, it just stopped accepting any new connection. Client used blocking connect to establish connections, and all these 10000 calls of connect returned success. Then everything freezes, no more packet(no retransmission), no more accepted connection.

But once I closed the client program, the server program started to accept the remaining connections after closing some connections (and eventually accepted all 10000 connections and closed all these connections).

When I changed backlog to 100 or larger, all 10000 connections were accepted without any problem. (so it is not fd resource problem)

I know when accepting queue is full, linux may just ignore incoming ACK of a 3-Way handshake, leaving client's connection established but server's connection still unestablished, then let the retransmit mechanism work. And eventually server retransmit SYN/ACK pakcet, client responds with ACK to re-establish this connection if server's accepting queue is available. If there is no available space in queue, server ignores the ACK again.

But when I use wireshark to monitor these retransmissions, I found that just a small number of SYN/ACK retransmissions occurred (like 100~200, much less than number of lost connections, which is between 500~1500), and they are retransmitted just once or twice, all less than the value specified in /proc/sys/net/ipv4/tcp_synack_retries. I inspected some of these retransmitted SYN/ACK packets, all of which received ACKs from client. But the number of retransmitted SYN packets from client was large.

So what's the underlying details?

Here are my codes: client server

Solution

  • When the backlog is full and SYN cookies are enabled, the kernel will activate the temporary SYN flood mode. This diagram shows how the cookies work (source) :

    enter image description here

    When SYN flood was activated, all ACK sent from the client will be dropped. The server sent only SYN/ACK and the cookie to the client, the cookie cache table will be much smaller than socket table to maintain actives connection. In this case, at the client side, the socket was considered as established but nothing is really opened at the server side (half-open connexion).

    When the client application closes the socket, a FIN packet with ACK flag set will be sent to the server with a cookie inside, the server sees that the connection has an ACK, at this point, if the backlog is not full, the server will try to rebuild the connection from the cookie. If the cookie is valid (within a valid round trip), the socket will be added to backlog queue and can be handled as a normal socket.

    It means that, when SYN cookie mode was activated and the backlog queue is not full, if the client sends some data to the server via the half-open sockets, accept() function will return new incomings connection.