Delphi Indy IdTcpClient read operation returning truncated data for one specific request

Question

This is an interesting problem that I’ve not been able to solve yet.

I am writing a client that communicates across the Internet to a server. I am using the TIdTcpClient Internet Direct (Indy) component in Indy 10 using RAD Studio 2007 native personality.

To get data from the server, I issue an HTTP request using SSL over port 443 where my request details are contained in the HTTP message body. So far, so good. The code works like charm, with one exception.

There is one request that I am submitting that should produce a response of about 336 KB from the server (the HTTP response header contains Content-Length: 344795). The problem is that I am getting only 320KB back. The response, which is in XML, is clearly truncated in the middle of an XML element.

For what it’s worth, the XML is simple text. There are no special characters that can account for the truncation. My TIdTcpClient component is simply reporting that, after receiving the partial response, that the server closed the connection gracefully (which is how every response is expected to be completed, even those that are not truncated, so this is not a problem).

I can make nearly identical calls to the same server where the response is also more than a few K bytes, and all of these work just fine. One request I make returns about 850 KB, another returns about 300 KB, and so on.

In short, I encounter this problem only with the one specific request. All other requests, of which there are many, receive a complete response.

I have talked to the creator of the service, and have supplied examples of my request. He reported that the request is correct. He also told me that when he issues my same request to his server that he gets a complete response.

I’m at a loss. Either the creator of the service is mistaken, and there is actually a problem with the response on that end, or there is something peculiar about my request.

Is there a solution here that I'm missing? Note that I’ve also used a number of other read mechanisms (ReadString, ReadStrings, ReadBytes, etc) and all produce the same result, a truncation of this one specific response at the 320KB mark.

The code is probably not relevant, but I’ll include it anyway. Sorry, but I cannot include the XML request, as it includes proprietary information. (ReadTimeout is set to 20 seconds, but the request returns in about 1 second, so it's not a timeout issue.)

function TClient.GetResponse(PayloadCDS: TClientDataSet): String;
var
  s: String;
begin
  try
    try
      s := GetBody(PayloadCDS);
      IdTcpClient1.Host := Host;
      IdTcpClient1.Port := StrToInt(Port);
      IdTcpClient1.ReadTimeout := ReadTimeout;
      IdTcpClient1.Connect;
      IdTcpClient1.IOHandler.LargeStream := True;
      //IdTcpClient1.IOHandler.RecvBufferSize := 2000000;
      IdTcpClient1.IOHandler.Write(s);
      Result := IdTcpClient1.IOHandler.AllData;
    except
      on E: EIdConnClosedGracefully do
      begin
         //eat the exception
      end;
      on e: Exception do
      begin
        raise;
      end;
    end;
  finally
    if IdTcpClient1.Connected then
      IdTcpClient1.Disconnect;
  end;
end;

Answer 1

As I mentioned in my original question, this connection uses SSL. This requires that you use an IdSSLIOHandlerSocketOpenSSL component, which you assign to the IOHandler property of the IdTcpClient component. All of this was in my original design, and as I mentioned, many of my requests were being responded to correctly.

Over the weekend I discovered another request which was returning an incomplete response. I captured that original HTTP request, and using a tool called SOAP UI, I submitted that request to the server. Using SOAP UI, the server returned a complete answer. Clearly, something was wrong with my client, and not the server.

I finally found the solution by just fiddling around with various properties. The property that finally corrrected the problem was in the SSLOptions property of the IdSSLIOHandlerSocketOpenSSL class. The default value of SSLOptions.Method is sslvSSLv2. When I changed Method to sslvSSLv23, all responses returned complete.

Why I was able to retrieve some responses in full and not all before I made this change is still a mystery to me. Nonetheless, setting IdSSLIOHandlerSocketOpenSSL.SSLOptions.Method to sslvSSLv23 solved my problem.

Thank you Remy Lebeau, for your suggestions.