Search code examples
csocketstcp

Sockets - max size for send and receive buffers

For learning purpose I have created two simple C programms to simulate the following sockets scenario: a client in the loop sends 256 bytes to the server(send function with MSG_DONTWAIT flag) but the server doesn't read the data (recv function). For simplicity the server doesn't send any other data to the client. I would like to test when send method will return an error. I use linux ss -nmtp command to see what happens with socket buffers. First the server receive buffer (Recv-Q) and skmem r parameter is growing:

State  Recv-Q   Send-Q         Local Address:Port          Peer Address:Port 

ESTAB  768      0              127.0.0.1:9002              127.0.0.1:34246    users:(("tcpServer",pid=16455,fd=4))
skmem:(r2048,rb131072,t0,tb2626560,f2048,w0,o0,bl0,d0)

ESTAB  0        0              127.0.0.1:34246             127.0.0.1:9002     users:(("tcpClient",pid=16482,fd=3))
skmem:(r0,rb131072,t0,tb2626560,f0,w0,o0,bl0,d0)

When the server Recv-Q reaches value 98816 the client Send-Q and skmem w start growing. When the client send function returns the error 11 (Resource temporarily unavailable) the buffers sizes looks this way:

State  Recv-Q   Send-Q         Local Address:Port          Peer Address:Port

ESTAB  98816    0                  127.0.0.1:9002             127.0.0.1:49744    users:(("tcpServer",pid=21644,fd=4))
skmem:(r128256,rb131072,t0,tb2626560,f2816,w0,o0,bl0,d16)

ESTAB  0        2553798            127.0.0.1:49744            127.0.0.1:9002     users:(("tcpClient",pid=22042,fd=3))
skmem:(r0,rb131072,t0,tb2626560,f570,w2653638,o0,bl0,d0)

Does skmem rb refer to the max receive buffer size and skmem tb refer to the max send buffer size ? If so, how to explain difference between Recv-Q and skmem rb and Send-Q and skmem tb?

Edit

skmem rb is the same as the default size returned by the following command:

sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096    131072  6291456

but skmem tb is different than the default size returned by this command:

sysctl net.ipv4.tcp_wmem
net.ipv4.tcp_wmem = 4096    16384   4194304

How to explain it ?

Solution

  • There are a lot of factors all coming together in your test. I'll try to walk through them. Much of this is documented in the man page for ss and tcp.

    • Send-Q is the amount of memory currently devoted to buffering outbound data on that socket.
    • Recv-Q is the amount currently buffering inbound data on that socket.
    • Skmem r is the total allocated memory for received in-flight packets.
    • Skmem w is the total allocated memory for sent in-flight packets.
    • net.ipv4.tcp_mem contains the low, pressure, and high page count limits on memory for the TCP stack. The default page size is 4,096 so to arrive at bytes you will need to multiply these by the page size.

    There are a lot of factors here, but ultimate the memory usage from all TCP sockets combined can not exceed tcp_mem's high setting. If you are currently exceeding the pressure setting, certain limitations may be imposed on connections that are memory heavy. In your case, this test would qualify for those.

    This test isn't usually compatible with most real world scenarios, as applications will generally back off long before they receive resource temporarily unavailable.