Does anyone know how to detect the end of a response to a CONNECT request? I've been writing my own HTTP stack and have noticed that most proxies seem respond to this request in different ways, with none of them appearing to conform to the RFC. Specifically RFC 2616 section 4.4:
Any response message which "MUST NOT" include a message-body (such as the 1xx, 204, and 304 responses and any response to a HEAD request) is always terminated by the first empty line after the header fields, regardless of the entity-header fields present in the message.
If a Transfer-Encoding header field (section 14.41) is present and has any value other than "identity", then the transfer-length is defined by use of the "chunked" transfer-coding (section 3.6), unless the message is terminated by closing the connection.
If a Content-Length header field (section 14.13) is present, its decimal value in OCTETs represents both the entity-length and the transfer-length. The Content-Length header field MUST NOT be sent if these two lengths are different (i.e., if a Transfer-Encoding header field is present). If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.
If the message uses the media type "multipart/byteranges", and the ransfer-length is not otherwise specified, then this self- elimiting media type defines the transfer-length. This media type UST NOT be used unless the sender knows that the recipient can arse it; the presence in a request of a Range header with ultiple byte- range specifiers from a 1.1 client implies that the lient can parse multipart/byteranges responses.
A range header might be forwarded by a 1.0 proxy that does not understand multipart/byteranges; in this case the server MUST delimit the message using methods defined in items 1,3 or 5 of this section.
By the server closing the connection. (Closing the connection cannot be used to indicate the end of a request body, since that would leave no possibility for the server to send back a response.)
Assuming the proxy supports the CONNECT method and aren't blacklisting the host/port I am trying to connect to, they return status code 200 which could in theory could contain a message body, so I can't use method 1. None of the proxies I've tested contained headers which would allow me to use methods 2-4. And because the connection has to stay open since the CONNECT method provides behavior similar to a SOCKS proxy, I can't use method 5. The closest I've seen is Fiddler (and possibly other) proxies send a Connection: close header even though they don't actually close the connection, since this is the closest thing they could do to support method #5. And some proxies don't even provide any headers, just a status line followed by a CR/LF.
Is there a special way you're supposed to handle these responses that is defined in an another RFC, or in a section which I've managed to overlook? Section 9.9 of the RFC which defines the CONNECT method is severely lacking in specifying how this method should be handled as it is only a single sentence.
So it looks like in the new proposed RFC they do define how CONNECT responses are to be handled. That they can not contain a message body, and you determine the end of the response by a empty line (CR/LF)
http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-26.html#CONNECT