I'm building a simple proxy server using Jetty's HttpClient. I am using Jetty version 9.3.10.v20160621 on Java 1.8.0_45.
I have a situation in which I do a GET on a resource which will return a response that is approximately 3.5M in size (I determined this using soapUI). Knowing that Jetty's default max response size is 2M, I do the following when I create the HttpClient instance.
HttpClient client = new HttpClient();
client.setResponseBufferSize(4194304);
client.start();
Later I perform a synchronous GET request like so:
System.out.println("response buffer size = " + client.getResponseBufferSize());
retVal = client.GET(uri);
The console log has the following:
response buffer size = 4194304
Nevertheless I get a java.util.concurrent.ExecutionException when I execute the GET() (stack trace below). I can only conclude that either (a) there is a bug in the setResponseBufferSize() method or, (b) the setResponseBufferSize() method doesn't do what the documentation says it does (set the response buffer size). Does anybody know what is up with this?
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Buffering capacity exceeded
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:652)
at org.eclipse.jetty.client.HttpClient.GET(HttpClient.java:343)
at oracle.paas.tools.sifter.proxy.ProxySession.get(ProxySession.java:106)
at oracle.paas.tools.sifter.proxy.Endpoint.doGet(Endpoint.java:75)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:837)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:248)
at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:272)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:326)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:256)
at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1584)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1332)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:158)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:119)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:115)
... 9 more
HttpClient.setResponseBufferSize(int) is for configuring the internal buffer size used to read individual buffers from the network.
It has no relationship to your Response body content size.
The situation you are dealing with is that the simplistic HttpClient.GET(uri) call will buffer up the response into a FutureResponseListener which is limited to 2MB of memory usage.
This limit is merely a side effect of the java Future concepts.
This is a shorthand / convenience method meant for small responses.
You have a response that is larger, so its on you to use the asynchronous features of HttpClient to read from the response content buffers/streams that HttpClient is managing, processing the data according to what you need it to do.
Example:
// In Initialization Code, start the client
HttpClient client = new HttpClient();
client.start();
// In code much later on, use the client
// Don't constantly start/stop the HttpClient
// Treat the HttpClient as a browser, and each newRequest() as
// a tab on that browser.
InputStreamResponseListener listener = new InputStreamResponseListener();
// Send asynchronously with the InputStreamResponseListener
client.newRequest(uri).send(listener);
// Call to the listener's get() blocks until the headers arrived
Response response = listener.get(5, TimeUnit.SECONDS);
// Now check the response information that arrived to decide whether to read the content
if (response.getStatus() == 200)
{
byte[] buffer = new byte[256];
try (InputStream input = listener.getInputStream())
{
while (true)
{
int read = input.read(buffer);
if (read < 0)
break;
// Do something with the bytes just read
}
}
}
else
{
response.abort(new Exception());
}
Check out the HttpClient Usage.java for more examples