Search code examples
javaservlet-filtersembedded-jettyservlet-3.0jetty-9

Calling startAsync() on a servlet request throws IllegalStateException in a filter chain terminated by Jetty 9's ProxyServlet


Environment:

  • In my environment, Kibana 4.5.2 is running behind a "reverse proxy servlet" which is created by extending Jetty's "ProxyServlet" class.
  • This is done so that the Kibana web interface can be accessed using the URL https://Jetty_Server_IP:8443/visual-analytics/proxy/... Requests for this URL are intercepted by the reverse proxy running in the Jetty Server and redirected to https://localhost:5601/... i.e., to the Kibana Server that is running in the same machine as the Jetty Server.
  • The Kibana Server then processes the request forwarded by the Jetty Server and returns the response back to the web browser.
  • NOTE: Jetty is running in "embedded mode" within my application.

Issue:

The "reverse proxy servlet" is mapped to the URL "/visual-analytics/proxy/*".

There is another "filter" that is mapped to the URL "/visual-analytics/proxy/elasticsearch/.kibana/search/*" in which a ContinuationListener is being used as indicated by the following code snippet:

    ContinuationSupport.getContinuation(myRequestWrapper).addContinuationListener(new ContinuationListener() {

                        @Override
                        public void onTimeout(Continuation continuation) {
                            logger.log(Level.WARNING, "Request timeout...");
                        }

                        @Override
                        public void onComplete(Continuation continuation) {

                            HttpServletResponse httpResponse = (HttpServletResponse)continuation.getServletResponse();
                            if (httpResponse.getStatus() == HttpServletResponse.SC_OK || httpResponse.getStatus() == HttpServletResponse.SC_CREATED ) {
                                   //some business logic
                            }

                        }
                    });

    chain.doFilter(myRequestWrapper, response);

The above ContinuationListener was working fine with Jetty version 8.1.15.v20140411 and the listener's onComplete() method was being called. But after upgrading the Jetty version to 9.3.14.v20161028, the ContinuationListener is no longer working i.e., neither the listener's onComplete() method nor the onTimeout() method is being called.

Since the ContinuationListener is no longer working, I replaced it with a "javax.servlet.AsyncListener" implementation as indicated by the following code snippet:

                AsyncContext asyncContext = myRequestWrapper.startAsync();
                asyncContext.addListener(new AsyncListener() {

                    @Override
                    public void onTimeout(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.WARNING, "Async timeout...");
                    }

                    @Override
                    public void onStartAsync(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.INFO, "Async start...");
                    }

                    @Override
                    public void onError(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.SEVERE, "Async error...");
                    }

                    @Override
                    public void onComplete(AsyncEvent event) throws IOException 
                    {
                          HttpServletResponse httpResponse = (HttpServletResponse) event.getSuppliedResponse();
                         //HttpServletResponse httpResponse = (HttpServletResponse) event.getAsyncContext().getResponse();
                        if (httpResponse.getStatus() == HttpServletResponse.SC_OK || httpResponse.getStatus() == HttpServletResponse.SC_CREATED ) {
                                    //some business logic
                        }
                    }
        }, myRequestWrapper, httpServletResponse);

        chain.doFilter(myRequestWrapper, response);

But adding the above "AsyncListener" implementation resulted in the following "IllegalStateException":

   https://164.99.175.139:8443/visual-analytics/proxy/elasticsearch/.kibana/search/test1?op_type=create
    java.lang.IllegalStateException: s=DISPATCHED i=true a=STARTED
            at org.eclipse.jetty.server.HttpChannelState.startAsync(HttpChannelState.java:264)
            at org.eclipse.jetty.server.Request.startAsync(Request.java:2235)
            at javax.servlet.ServletRequestWrapper.startAsync(ServletRequestWrapper.java:432)
            at javax.servlet.ServletRequestWrapper.startAsync(ServletRequestWrapper.java:432)
            at javax.servlet.ServletRequestWrapper.startAsync(ServletRequestWrapper.java:432)
            at org.eclipse.jetty.proxy.ProxyServlet.service(ProxyServlet.java:88)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
            at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1712)
            at com.netiq.sentinel.kibana.search.KibanaSearchFilter.doFilter(KibanaSearchFilter.java:249)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
            at com.netiq.sentinel.kibana.proxy.AcceptEncodingHeaderModificationFilter.doFilter(AcceptEncodingHeaderModificationFilter.java:37)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
            at com.netiq.sentinel.kibana.proxy.SecurityFilter.doFilter(SecurityFilter.java:41)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
            at com.netiq.sentinel.kibana.proxy.AuditFilter.doFilter(AuditFilter.java:104)
            at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
            at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
            at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
            at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
            at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
            at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
            at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
            at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
            at org.eclipse.jetty.server.Server.handle(Server.java:534)
            at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
            at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
            at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
            at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
            at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
            at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
            at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
            at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
            at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
            at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
            at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
            at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
            at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
            at java.lang.Thread.run(Thread.java:745)

Line number 88 in "ProxyServlet.java" which is mentioned in the above stack trace has the following code:

final AsyncContext asyncContext = request.startAsync();

Is the "IllegalStateException" being thrown because the startAsync() method is being called twice on the same request - for the first time in my filter and for the second time in Jetty ProxyServlet's service() method? If yes, is there no way to use an "AsyncListener" in a filter chain that is terminated by Jetty's ProxyServlet?

Any pointers on how to proceed further would be greatly appreciated. Looking forward for any response...


Solution

  • FYI I fixed this issue as follows:

    1. Instead of using Jetty's "ContinuationListener", I used Servlet 3.0's "AsyncListener", since Jetty's "ContinuationListener" does not seem to work after upgrading to Jetty 9.3.14.v20161028. This may be a Jetty bug.
    2. The listener related code segment was located BEFORE the filter’s “chain.doFilter()” call. I moved this code segment to a position AFTER the filter’s "chain.doFilter()" call, so that I can retrieve the AsyncContext that is "already created" in the Jetty ProxyServlet's service() method. I can then add an AsyncListener to the retrieved AsyncContext.
    3. I changed the “request.startAsync()” call in my filter to “request.getAsyncContext()” so that I am not starting a new AsyncContext which leads to the IllegalStateException, but only retrieving the AsyncContext that is already created in Jetty’s ProxyServlet.

    So the updated code segment looks like this:

                chain.doFilter(myRequestWrapper, response);
    
                AsyncContext asyncContext = myRequestWrapper.getAsyncContext();
                asyncContext.addListener(new AsyncListener() {
    
                    @Override
                    public void onTimeout(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.WARNING, "Async timeout...");
                    }
    
                    @Override
                    public void onStartAsync(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.INFO, "Async start...");
                    }
    
                    @Override
                    public void onError(AsyncEvent event) throws IOException 
                    {
                                logger.log(Level.SEVERE, "Async error...");
                    }
    
                    @Override
                    public void onComplete(AsyncEvent event) throws IOException 
                    {
                          HttpServletResponse httpResponse = (HttpServletResponse) event.getSuppliedResponse();
                         //HttpServletResponse httpResponse = (HttpServletResponse) event.getAsyncContext().getResponse();
                        if (httpResponse.getStatus() == HttpServletResponse.SC_OK || httpResponse.getStatus() == HttpServletResponse.SC_CREATED ) {
                                    //some business logic
                        }
                    }
        }, myRequestWrapper, httpServletResponse);