AJP run out of thread

I have the two apache load balance proxy servers(named A1, A2) run with mod_jk, and two Jboss web servers(named J1, J2).

Jboss version is jboss-eap-5.1. Apache version is 2.2.15. OS: Red Hat Enterprise Linux Server release 6.4 (Santiago) Both A1 and A2 colud redirect the request(with port 80) from outside to J1, J2(both are port 8080). The setting of A1,A2 are the same(except HostIP). J1,J2 have the same setting too.(except HostIP and jvmRoute)

I have some problems with this architecture. J1 and J2 can process A1's request well(recive a request -> create a thread to process -> process finish -> close thread).

But A2's request is weird. When the process finished, the thread didn't closed. And network connection stay in ESTABLISHED. I used jstack PID to check, and there's a lot of following statements:

"ajp-XX.XX.72.12-8009-240" daemon prio=10 tid=0x00007f791418e800 nid=0x7793 runnable [0x00007f7875d5c000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1036)
    at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1115)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:383)
    at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:385)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
    at java.lang.Thread.run(Thread.java:662)`

Compared the above PID(nid=0x7793 ) with ps -efT | grep 30611 , can get the following: `xxxx 17869 30611 17821 0 20:40 ? 00:00:00 /usr/local/java/bin/java -program.name=run.sh ...

And while I use netstat -A inet, there are lots of connection like this:

Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 npouiap2:8009               10.100.84.34:49666          ESTABLISHED 
tcp        0      0 npouiap2:8009               10.100.84.34:50374          ESTABLISHED 
tcp        0      0 npouiap2:8009               10.100.84.34:50507          ESTABLISHED

Solution

The problem may be the threads aren't idle as AJP connections are persistent by default.

You can try some one of these options:

Increase maxThreads on JBoss to be the same as "max threads" (exact directives vary with MPM) on httpd
Turn on mod_jk's disable reuse option

The option 2 is the simplest to apply and verify if the problem persist.