Tomcat shutdown VERY slow after calling shutdown.sh?

My situation is like this:

Everytime before uploading war file to web-app folder, I stop Tomcat by calling sh shutdown.sh. It used to take about 30 seconds for a total shutdown. But now it doesn't work well anymore.

Actually, it did some work, because when I access the application from web-page it throws 503 error (Under Maintenance). But when I use ps aux | grep tomcat to check, the tomcat process is still there. And it will be there for around 5 - 10 mins.

I understand that it may need to take extra times to complete all the tasks, but it is way too slow (5 - 10 minutes), before it is stop totally. I don't understand why this happens, but there must be some reason. Maybe there's something to do with the code, or the new script of deployment we used recently. I just have almost no clue about where to check.

This is important to our team because we are using "auto-deployment", in which we use a script to auto-package war file, uploading and deploy on a specific time. If we started a new tomcat instance before the old one successfully shutdown, it will hang there for eternal, and cleaning up task by "kill -9" is daunting.

Is there anyone who has experimented this issue? Any clue would be appreciated.

Solution

The fact that you see your Quartz jobs running, and the error message, are both significant:

SEVERE: The web application [/project] appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak.

One suggestion is configuration:

http://forum.springsource.org/showthread.php?17833-Spring-Quartz-Tomcat-no-shutdown

I had the same problem. I fixed it by adding destroy-method="destroy" to the SchedulerFactoryBean definition. This way spring closes down the scheduler when the application is stopped.

Another suggestion is to add a shutdown listener:

http://forums.terracotta.org/forums/posts/list/15/4341.page

Using a context listener and introducing a timeout on shutdown solves the issue for me. I just wait a second after shutting down:
  public void contextDestroyed(ServletContextEvent sce) {
    try {
      factory.getScheduler().shutdown();
      Thread.sleep(1000);