How to determine when data is finished on an SSL socket

Question

I'm writing an application that creates multiple non-blocking SSL connections to an https server, in each one I send a request and read the server's response. my problem is, whatever I do, I can't determine when the response data is finished. here's part of my code responsible for sending and receiving data:

....
fd_set connectionfds;
struct timeval timeout2;
FD_ZERO(&connectionfds);
FD_SET(socket_server, &connectionfds);
timeout2.tv_usec = 0;
timeout2.tv_sec = 1;
while(1)
{
    r=BIO_read(io,buf,BUFSIZZ-1);
    if (r>0){
        //gather data
        continue;
    }
    else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
        int ret = select(socket_server + 1, &connectionfds, NULL, NULL, &timeout2);
        if (ret <= 0){
            break;
        }
        continue;
    }
    else{
        break;
    }
}
// use whole gathered data
....

my problem with above code is, if I set select timeout to a small time, I can't guarantee that all data is received (because some servers are really slow), and if I set timeout to a long time (5-10 seconds), my socket is getting stucked in wait-state for a long time and I can't use the response before that. I tried to make this work by using "BIO_should_read()" or "BIO_pending()" functions, but none of them is giving me what I want. so, Is there a way to determine when exactly there is nothing else to read on the SSL socket?

Answer 1

As Steffen said, there is no notion of "end-of-data" in SSL and TCP/IP stream. Modern http servers are using a "keep-alive" connection and do not disconnect after answering your request. Instead they are using additional info from where you need to infer the length of the answer. It is either a "Content-lenght" header, or they use "Transfer-Encoding:chunked" followed by chunks of data. Each chunk is preceded by size (as hexadecimal number). A chunk with the size equal to zero means the end of the answer.

For you, the easiest way to get something usable may be to send request as a HTTP 1.0 request (not HTTP 1.1). If you issue a request like this:

GET /myrequestdir/myrequestfile HTTP/1.0

the server will close the connection right after answering your request.

In case you need HTTP 1.1, here are two examples, how the server can answer with a "Hello." message:

HTTP/1.1 200 OK
Content-Length:6

Hello.

or with a chunked connection:

HTTP/1.1 200 OK
Transfer-Encoding: chunked

6
Hello.

0