Slow reading of HTTP response using InputStreamReader and BufferedReader?

Question

I have the following code in Java that sends an HTTP request to a web server and read the response:

StringBuilder response = new StringBuilder(50000);
URL url2 = new URL(ServiceURL); 
connection = (HttpURLConnection)url2.openConnection();
connection.setRequestMethod("POST");

//... (some more connection settings) ...

BufferedWriter  wr = new BufferedWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

wr.write(Request);
wr.flush ();
wr.close ();

InputStream is = connection.getInputStream();       
BufferedReader rd = new BufferedReader(new InputStreamReader(is));

int i = 0;
while ((i = rd.read()) > 0) {
    response.append((char)i);
}

It works for most cases, but I have a problem with one server that returns a rather large XML (something like 500KB; I guess this is pretty large for just a bunch of text..), where I keep getting a read timeout exception.

I believe it's not a network problem, because I've tried making the same request using curl and the response just arrived all right and pretty quick, something like two seconds.

When I look what's going on in the network (using wireshark to capture the packets), I noticed that the TCP receive window in my computer gets full at some point. The TCP stack sometimes survives this; I can see the server sending TCP keep-alive to keep the connection up, but in the end the TCP connection just breaks down.

Could it be that the reading part of the code (appending the received response character-by-character) is slowing my code down? Is there a more efficient way to read an HTTP response?

Answer 1

Reading character by character is quite slow, yes. Try reading chunks at a time into a buffer:

char[] buf = new char[2048];
int charsRead;
while((charsRead = rd.read(buf, 0, 2048)) > 0) {
    response.append(buf, 0, charsRead);
}