Our software (Nmap port scanner) needs to quickly determine the status of a non-blocking TCP socket connect(). We use select() to monitor a lot of sockets, and Windows is good at notifying us when one succeeds. But if the port is closed and the target sends a TCP RST, Windows will keep trying a few times before notifying the exceptfds, and the socket error is WSAECONNREFUSED as expected. Our application has its own timeout, though, and will usually mark the connection as timed-out before Windows gives up. We want to get as close as possible to the behavior of Linux, which is to notify with ECONNREFUSED immediately upon receipt of the first RST.
We have tried using the TCP_MAXRT socket option, and this works to get select() to signal us right away, but the result (for closed ports) is always WSAETIMEDOUT, which makes it impossible to distinguish closed (RST) from filtered/firewalled (network timeout), which puts us back at the original problem. Determining this distinction is a core feature of our application.
So what is the best way on Windows to find out if a non-blocking socket connect() has received a connection reset?
EDITED TO ADD: A core problem here is this line from Microsoft's documentation on the SO_ERROR socket option: "This per-socket error code is not always immediately set." If it were immediately set, we could check for it prior to the connect timeout.
After several years, we found the correct combination of socket options and ioctls to accomplish this:
TCP_MAXRT or (since Windows 10 1607) TCP_MAXRTMS to set the connection timeout, after which time the socket error is set to WSAETIMEDOUT.WSAIoctl() to set the number of SYN retries and the initial retransmission timeout, which is doubled for each retry to get the next timeout. The control code is SIO_TCP_INITIAL_RTO.When a connection is attempted, Winsock begins sending SYN packets. If the first receives a RST, another SYN is sent after a brief wait. If instead it times out, the retransmission timeout is increased and another SYN is sent. When the last SYN retry has finished, the socket error code will be set to WSAECONNREFUSED if the response was RST or WSAETIMEDOUT if there was no response. If the connection timeout happens before the last SYN retry has finished, the socket error code is WSAETIMEDOUT regardless of whether any response was RST.
To allow WSAECONNREFUSED to be set, the connection timeout has to be long enough to allow all the SYN retries to be sent. In most cases, each SYN will get a RST, so the time required is just that small wait (half a second, if I recall correctly) times the number of retries. However, it is possible for the first SYN attempts to time out due to packet loss, then a RST to arrive for the last attempt, so some calculation is necessary to accommodate all situations.