I've been using the Java Service wrapper in a custom application for quite a while and it's been working fine. Since updating our application to a new version in the last few days the JVM started hanging and then wrapper prints this in the log: JVM appears hung: Timed out waiting for signal from JVM.
It then automatically terminates the JVM and starts the app again. This happens after about 10 hours of running, which just makes it harder to debug.
Of course I am going to look through the changes that we've made, but no major changes were made that I would suspect is causing this type of problem.
Where can I look to try and figure out what is happening? Debug messages from the application don't indicate anything interesting. If the JVM just crashes it will usually create a dump, which can help in debugging it, but it's hanging, so it's not creating a dump. If I make it not restart the service automatically is there anything I can do to get some useful information out of the JVM before restarting it?
It seems to me that the JVM shouldn't hang from typical programming errors. What have you run into before that can cause the JVM to hang?
I had a couple different versions of a library on the classpath (JBPM). With wrapper you can use wildcards to include jars. Be careful with this though as you may accidentally include more than you should.
Here is an IBM article that gives information on debugging hangs in Java. It basically says that there are two things that can cause hangs:
Since then I've had to debug other hanging issues. On linux you can send the JVM the QUIT signal to make it do a thread dump to the console. This really helps figuring out where the issue is. Use this command to do that: kill -QUIT
Edit 6/13/2017
These days I use jmap included in the JDK to dump the entire memory of the program. Then I use Eclipse Memory Analyzer to see the exact state of the program when it crashed. You can look at the list of threads that are active and then inspect the variables in each stack frame.
/usr/java/latest/bin/jmap -dump:file=/tmp/app-crash.hprof <PID>
Where PID is the process ID of the java process.