I have a JSP web-app hosted on IIS 7.5 & Tomcat 8.0.30 via BonCode AJP13 v1.0.26 on Win 2008 R2 64 bit.
This has been working OK for a few months to date.
We have identical environments: development, staging & production.
On the development and staging environments, suddenly, we are getting very slow responses and in some cases, intermittent IIS generated timeout errors:
Request timed out
Exception Details: System.Web.HttpException: Request timed out.
[HttpException (0x80004005): Request timed out.]
On the production environment, performance seems OK.
Tried re-booting servers, restarting IIS, restarting Tomcat etc
Has anyone had these slow-downs?
Does anyone use a known working version of BonCode greater than 1.0.26?
Details
Bypassing IIS & BonCode, ie explicity selecting port 8080, gives an instant response.
http://<domain>:8080/<path>
The slowdown is not JSP page specific. Even very simple pages, eg get date & time, still intermittently gives slow responses.
There is no authentication on Tomcat / JSP web-app.
A single instance of Tomcat residing on the same server as IIS 7.5 ie no load-balancing etc.
Only plain HTTP is being used ie not HTTPS and no associated certificates.
Server Diagnosis
Occasionally, this is seen:
2016-02-29 15:59:01 1.0.26 ERROR
TCP Client level -- Server/Port:localhost/8009
Value cannot be null.
Parameter name: buffer
2016-02-29 15:59:01 One Connection raised an error
but usually, no related errors / warnings are being shown
Comparing with older logs
Old log
2017-05-04 10:19:28 New Connection 46 of 0 to tomcat: 127.0.0.1:8009 ID: 46 [T-5]
2017-05-04 10:19:35 New Connection 47 of 0 to tomcat: 127.0.0.1:8009 ID: 47 [T-45]
2017-05-04 10:25:11 New Connection 48 of 0 to tomcat: 127.0.0.1:8009 ID: 48 [T-45]
2017-05-04 10:25:15 New Connection 49 of 0 to tomcat: 127.0.0.1:8009 ID: 49 [T-49]
2017-05-04 10:39:12 New Connection 50 of 0 to tomcat: 127.0.0.1:8009 ID: 50 [T-49]
2017-05-04 10:40:09 New Connection 51 of 0 to tomcat: 127.0.0.1:8009 ID: 51 [T-22]
2017-05-04 10:41:29 New Connection 52 of 0 to tomcat: 127.0.0.1:8009 ID: 52 [T-49]
2017-05-04 10:42:03 New Connection 53 of 0 to tomcat: 127.0.0.1:8009 ID: 53 [T-52]
2017-05-04 11:05:04 New Connection 1 of 0 to tomcat: 127.0.0.1:8009 ID: 1 [T-20]
2017-05-04 11:06:37 New Connection 2 of 0 to tomcat: 127.0.0.1:8009 ID: 2 [T-23]
2017-05-04 11:07:23 New Connection 3 of 0 to tomcat: 127.0.0.1:8009 ID: 3 [T-51]
2017-05-04 11:08:55 New Connection 4 of 0 to tomcat: 127.0.0.1:8009 ID: 4 [T-23]
2017-05-04 11:09:17 New Connection 5 of 0 to tomcat: 127.0.0.1:8009 ID: 5 [T-51]
New log
2017-05-16 14:31:57 New Connection 200 of 0 to tomcat: 127.0.0.1:8009 ID: 200 [T-153]
2017-05-16 14:31:58 New Connection 201 of 0 to tomcat: 127.0.0.1:8009 ID: 201 [T-153]
2017-05-16 14:32:30 New Connection 202 of 0 to tomcat: 127.0.0.1:8009 ID: 202 [T-153]
2017-05-16 14:36:05 New Connection 204 of 0 to tomcat: 127.0.0.1:8009 ID: 204 [T-153]
2017-05-16 14:36:06 New Connection 205 of 0 to tomcat: 127.0.0.1:8009 ID: 205 [T-153]
2017-05-16 14:41:38 New Connection 206 of 0 to tomcat: 127.0.0.1:8009 ID: 206 [T-153]
2017-05-16 14:42:51 New Connection 207 of 0 to tomcat: 127.0.0.1:8009 ID: 207 [T-153]
2017-05-16 14:42:52 New Connection 208 of 0 to tomcat: 127.0.0.1:8009 ID: 208 [T-153]
2017-05-16 14:43:12 New Connection 209 of 0 to tomcat: 127.0.0.1:8009 ID: 209 [T-153]
2017-05-16 14:43:40 New Connection 210 of 0 to tomcat: 127.0.0.1:8009 ID: 210 [T-153]
In the old log, the connection ids seem to reset after ~20-80 with each connection using a range of threads, occasionally the same thread.
In the new log, with the slow down, the connection ids are far higher and seem to be using the same thread.
no errors, as expected as servlet/JSP requests are not being forwarded.
Configuration Files
BonCode - BonCodeAJP13.settings
<Settings>
<Port>8009</Port>
<Server>localhost</Server>
<MaxConnections>0</MaxConnections>
<LogLevel>2</LogLevel>
<LogDir>E:\inetpub\logs\BonCode</LogDir>
<FlushThreshold>0</FlushThreshold>
<EnableRemoteAdmin>False</EnableRemoteAdmin>
<HeaderBlacklist>AUTHORIZATION,URL,SERVER_SOFTWARE,SERVER_NAME,SERVER_PROTOCOL</HeaderBlacklist>
<PacketSize>65536</PacketSize>
</Settings>
Tomcat - server.xml
...
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
packetSize="65536" />
...
Notes
I am aware that BonCode v1.0.26 has now been updated to v1.0.36, but the newer version has only been out for about 10 days.
If I do not get anywhere, I will upgrade but there are limits to what I can do on the production environment.
Long term: plan for upgrade. You are back 10 versions.
Most likely you are exhausting the number of AJP connections. I would adjust the Tomcat connection and timeout parameters to remove stuck connections:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" keepAliveTimeout="-1" connectionTimeout="60000" maxThreads="1000" packetSize="65536" />
Best, Brian