I am setting up an automated Cron job on a bunch of Linux servers which requires invoking some PUT/POST APIs provided by an IBM WebSphere Application Server(8.5.5). I wrote my own REST client in Java, but strangely sometimes the API execution hangs with below exception.
com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[JAX-RS Servlet]: java.net.SocketTimeoutException: Async operation timed out
at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:191)
at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4212)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3440)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3546)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1822)
at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:371)
at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:265)
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:322)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
at java.io.InputStreamReader.read(InputStreamReader.java:188)
at com.ibm.json.java.JSONArray.parse(JSONArray.java:150)
at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:44)
at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:25)
at org.apache.wink.server.internal.registry.ServerInjectableFactory$EntityParam.getValue(ServerInjectableFactory.java:197)
at org.apache.wink.common.internal.registry.InjectableFactory.instantiate(InjectableFactory.java:68)
at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:44)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.OptionsMethodWADLHandler.handleRequest(OptionsMethodWADLHandler.java:51)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:60)
at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)
at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)
at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:184)
... 78 more
This only occurred on some of the Linux servers, and it worked perfectly fine when I developed it in my Windows workstation. So it shouldn't be a problem with time out settings in the WAS server. I tried to debug using curl as below
curl -vX PUT http://<was_server_ip>:9081/RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen -d @test.json --header "Content-Type: application/json"
and found out when the payload exceeds 1306 bytes (see Content-Length in curl output below), the API starts hanging and gives above-mentioned exception.
> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen
HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1307
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: text/html;charset=ISO-8859-1
< $WSEP:
< Content-Language: en-US
< Content-Length: 102
< Connection: Close
<
Error 500: javax.servlet.ServletException: java.net.SocketTimeoutException: Async operation timed out
* Closing connection #0
When the payload is less than 1307 bytes, the API gets executed properly.
> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1306
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< HTTP/1.1 200 OK
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: application/json
< Access-Control-Allow-Headers: application/json
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE
< Access-Control-Allow-Origin: *
< Content-Language: en-US
< Transfer-Encoding: chunked
<
* Connection #0 to host 9.51.163.190 left intact
* Closing connection #0
{"Identifier":"syncTestSuites completed","Result":{"Newly Added Test Suites":[]}}
I need to check with their administrators on what settings might cause this on those problematic Linux machines. But also I am posting it here to see if anyone could give some thoughtful insight on the issue.
The problem was resolved long ago but forgot to update it here.
After the MTU size of the WAS application server was lowered from 1500 bits to 1320, the problem was gone. According to the network admin, this made the data segment of the server a bit smaller, allowing some applications to handle the data in a better way.
Here is a very good blog article on Maximum Transmission Unit https://craigocon.wordpress.com/2012/10/03/whats-mtu-and-how-do-i-change-it-in-windows-2008-r2/.
I am completely new to network knowledge, hence glad that I got support form our network team.