I'm writing an HTTP server on a very limited network device which uses uIP 1.0 as a TCP/IP stack. This library doesn't buffer any data, it only calls the user callback to provide a single packet to be sent or to re-issue previous data for retransmission. Next data packet is sent after the previous one has been ACKed by the remote side. First packet sent by my server is the response headers (45 bytes), then the file is sent in chunks of 846 bytes. The problem is that the remote side (macos or Linux, firefox or wget) stops sending ACKs starting with the third packet. Not getting any ACKs, my device starts retransmitting that packet and eventually resets the connection. This happens every time in exactly the same way. If I divide the data in smaller chunks (say, 256 bytes each), more packets get ACKed before ACKs stop coming - around 10 instead of 3.
Here is the dump of one session, the dump was performed on the client site (192.168.178.30) using wireshark:
1 0.000000 192.168.178.30 192.168.178.70 TCP 78 56391 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=3618183324 TSecr=0 SACK_PERM
2 1.001115 192.168.178.30 192.168.178.70 TCP 78 [TCP Retransmission] 56391 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=3618184325 TSecr=0 SACK_PERM
3 1.096615 192.168.178.70 192.168.178.30 TCP 1008 80 → 56391 [SYN, ACK] Seq=0 Ack=1 Win=846 Len=0 MSS=846
4 1.097032 192.168.178.30 192.168.178.70 TCP 54 56391 → 80 [ACK] Seq=1 Ack=1 Win=65535 Len=0
5 1.125337 192.168.178.30 192.168.178.70 TCP 192 56391 → 80 [PSH, ACK] Seq=1 Ack=1 Win=65535 Len=138
6 1.447170 192.168.178.70 192.168.178.30 TCP 1008 80 → 56391 [PSH, ACK] Seq=1 Ack=139 Win=846 Len=45
7 1.447485 192.168.178.30 192.168.178.70 TCP 54 56391 → 80 [ACK] Seq=139 Ack=46 Win=65535 Len=0
8 1.754454 192.168.178.70 192.168.178.30 TCP 1008 80 → 56391 [PSH, ACK] Seq=46 Ack=139 Win=846 Len=846
9 1.754807 192.168.178.30 192.168.178.70 TCP 54 56391 → 80 [ACK] Seq=139 Ack=892 Win=65535 Len=0
10 2.061660 192.168.178.70 192.168.178.30 TCP 1008 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
11 2.794855 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
12 3.597716 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
13 4.645639 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
14 6.567428 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
15 9.844375 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
16 15.988703 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
17 22.132761 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
18 28.277190 192.168.178.70 192.168.178.30 TCP 1008 [TCP Retransmission] 80 → 56391 [PSH, ACK] Seq=892 Ack=139 Win=846 Len=846
19 34.318832 192.168.178.70 192.168.178.30 TCP 1008 80 → 56391 [RST, ACK] Seq=892 Ack=139 Win=846 Len=0
The packets which didn't receive the ACK had incorrect TCP checksum. Wireshark does not verify TCP checksum by default, that's why I didn't notice this problem.