I have two java webapps, each one deployed on a separate tomcat server (and in a separate machine), one of them makes a lot of consecutive (but sequential, not parallel) http requests to the other one (+1000). It works most of the time, however when the load on the "client server" (the one that generates the requests) is high, I get random IO errors I have been unable to track down.
Here's a simplified version of the client
public class BPXProxy {
private static HttpClient client;
public static void init(){
client = new HttpClient();
}
public static InvoiceCreationInfo registerInvoice(InvoiceData invoice) throws SQLException, HttpException, IOException {
JSONObject invoiceJSON = new JSONObject();
invoiceJSON.put("invoicer", invoice.getInvoicerIdentification().getDocument());
invoiceJSON.put("buyer", invoice.getBuyerIdentification().getDocument());
String serviceURL = ConfigurationManager.getBPXServicesPath() + "Invoice?operation=registerExternalInvoice&branchCode="+ branchCode;
PostMethod updatePage = new PostMethod(serviceURL);
updatePage.addParameter("invoice", invoiceJSON.toJSONString());
updatePage.addParameter("invoiceSubject", invoice.getInvoiceSubject());
client.executeMethod(updatePage); // java.io.IOException: Stream closed, java.net.SocketException: Socket closed
String response = updatePage.getResponseBodyAsString(); // java.io.IOException: chunked stream ended unexpectedly, java.io.IOException: CRLF expected at end of chunk: -1/-1
updatePage.releaseConnection();
JSONObject jsonResponse = JSONSimpleHelper.parseJSON(response);
return jsonResponse;
}
}
Examples of these exceptions:
java.io.IOException: chunked stream ended unexpectedly
at org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:251)
at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:220)
at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:175)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:107)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:126)
at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:684)
at org.apache.commons.httpclient.HttpMethodBase.getResponseBodyAsString(HttpMethodBase.java:735)
at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:208)
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:206)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
at org.apache.commons.httpclient.HttpParser.parseHeaders(HttpParser.java:165)
at org.apache.commons.httpclient.HttpMethodBase.readResponseHeaders(HttpMethodBase.java:1790)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1592)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:207)
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.getAccountStatusInvoices(BPXProxy.java:2905)
java.io.IOException: CRLF expected at end of chunk: -1/-1
at org.apache.commons.httpclient.ChunkedInputStream.readCRLF(ChunkedInputStream.java:206)
at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:218)
at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:175)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:107)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:126)
at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:684)
at org.apache.commons.httpclient.HttpMethodBase.getResponseBodyAsString(HttpMethodBase.java:735)
at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.getPersonOpenReceivables(BPXProxy.java:3132)
org.apache.commons.httpclient.ProtocolException: The server ec2-54-196-218-148.compute-1.amazonaws.com failed to respond with a valid HTTP response
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1846)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:207)
Any idea why these could happen? does it look like the fix should be for the "client server" or the other server?
I am doing this on aws, each server is on a separate ec2 node, each request is taking about 4 seconds to complete. I'm using apache commons httpclient to make the calls; I have monitored the servers and there are no issues with memory consumption, and I don't see peaks with network in and network out on each servers.
I recommend two things.
First, use instance fields and methods instead of static fields and methods. You might have a race condition because the responses are not coming in as you expect, and you're closing streams in one invocation of registerInvoice and then trying to read from them in another invocation. So, your class would look like
public class BPXProxy {
private HttpClient client;
public void init(){
client = new HttpClient();
}
public InvoiceCreationInfo registerInvoice(InvoiceData invoice) throws SQLException, HttpException, IOException {
JSONObject invoiceJSON = new JSONObject();
invoiceJSON.put("invoicer", invoice.getInvoicerIdentification().getDocument());
invoiceJSON.put("buyer", invoice.getBuyerIdentification().getDocument());
String serviceURL = ConfigurationManager.getBPXServicesPath() + "Invoice?operation=registerExternalInvoice&branchCode="+ branchCode;
PostMethod updatePage = new PostMethod(serviceURL);
updatePage.addParameter("invoice", invoiceJSON.toJSONString());
updatePage.addParameter("invoiceSubject", invoice.getInvoiceSubject());
client.executeMethod(updatePage); // java.io.IOException: Stream closed, java.net.SocketException: Socket closed
String response = updatePage.getResponseBodyAsString(); // java.io.IOException: chunked stream ended unexpectedly, java.io.IOException: CRLF expected at end of chunk: -1/-1
updatePage.releaseConnection();
JSONObject jsonResponse = JSONSimpleHelper.parseJSON(response);
return jsonResponse;
}
}
and you would invoke it like
BPXProxy instance = new BPXProxy();
instance.init();
InvoiceCreationInfo info = instance.registerInvoice(invoice);
Second, it looks like you're using Apache HttpClient 3.x, which is severely outdated. Upgrade to version 4.3 (or later if it exists). You'll have to change around your code, but based on the quick start, it shouldn't be too drastic:
public class BPXProxy {
private CloseableHttpClient client = HttpClients.createDefault();
public InvoiceCreationInfo registerInvoice(InvoiceData invoice) throws SQLException, HttpException, IOException {
JSONObject invoiceJSON = new JSONObject();
invoiceJSON.put("invoicer", invoice.getInvoicerIdentification().getDocument());
invoiceJSON.put("buyer", invoice.getBuyerIdentification().getDocument());
String serviceURL = ConfigurationManager.getBPXServicesPath() + "Invoice?operation=registerExternalInvoice&branchCode="+ branchCode;
HttpPost httpPost = new HttpPost(serviceURL);
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add("invoice", invoiceJSON.toJSONString());
nvps.add("invoiceSubject", invoice.getInvoiceSubject());
client.setEntity(new UrlEncodedFormEntity(nvps));
CloseableHttpResponse rsp = client.execute(httpPost);
try {
HttpEntity entity = rsp.getEntity();
String rspJson = EntityUtils.toString(entity);
EntityUtils.consume(entity);
return JSONSimpleHelper.parseJSON(rspJson);
} finally {
rsp.close();
}
}
}
(Warning: I have not tried to compile or test this code.)