I found the request from client timeout sometimes. And I use tcpdump on the server side to catch some special tcp packet as follows. It seems no packet was lost. Why the server doesn't reply to client after receiving the first SYN? enter image description here
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
SYN_RECV 220
CLOSE_WAIT 1
ESTABLISHED 496
FIN_WAIT1 42
FIN_WAIT2 2
TIME_WAIT 72588
sysctl -a | grep tcp
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 0
net.ipv4.tcp_fastopen_key = fc1980cc-2a3369bf-91223671-bb8e012f
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_init_cwnd = 10
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_loss_init_cwnd = 1
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 524288
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 524288
net.ipv4.tcp_mem = 761319 1015094 1522638
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_delay_ack = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_ignore_syn_tsval_zero = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_recycle_private_only = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_workaround_signed_windows = 0
I had try to set tcp_tw_reuse tcp_tw_recycle and increase tcp_max_syn_backlog .There still error: error=HTTP 599: Timeout while connecting,headers=tornado.httputil.HTTPHeaders object at 0x7f74ed884860,reason='Unknown'
Maybe the TCP connection backlog of the server is full.
man page of listen says:
if the underlying protocol supports retransmission, the request may be ignored so that a later reattempt at connection succeeds.
Try to increase backlog parameter value of listen() call of the server.