Today in the company where I work we received a report about one of our webapps not working.
The first thing we did was look at the hardware utilization:
Processor: 5%;
Memory: 68%;
Disk IO capacity: 4%;
Network: 1Mbps/1Gbps;
After some tests we saw that, indeed, the webapp pages weren't loading, some time after the webapp timeouts.
Other webapps in the exactly same tomcat instance were working fine and fast as ever, no problem with them. We tried to restart/reload the webapp but it still didn't get it working. Finally we restarted tomcat which corrected the issue for now.
There were no restarts or redeploys since yesterday when it was working fine. We believe this may be a periodic bug of some sort so we want to correct it soon.
Does anyone know of any steps we may take to investigate what it might have been?
It seems related to lock on IO.
If you are using linux, the first thing is do is to check open files by using the command lsof . If you see a lot of opened files by your app, you have to check in the code that every InputStream/OutputStream are closed (even in exception handling code).
An other common source of issues is related to thread starvation, see http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve