How to kill a child process spawned by Java when tomcat is shutdown

I have written a service for JIRA(a web application runs in tomcat) which runs periodically(say 1 hour). Basically, the service executes a system command thru runtime.exec(command) and parses the output generated by the command then updates a Lucene index with it, output will be huge.

The problems are:

1) If I shutdown tomcat with shutdown.sh while the above service is executing, the java(or the catalina) process is not getting killed. Both the java & child process are living for a while i.e., until the system command completes & service processes the output. But then the service fails to update the index leaving the index in an inconsistent state.

If I shutdown tomcat when the above service is not running, everything is good. I think, this is explained here. I am still not clear why JVM won't shutdown as the above service is running within tomcat?

Note that this is the only java app running on that machine.

2) Then, if I kill java using kill <pid>, both the java & child process are getting killed contradicting to this post.

Is this because the child process is sending output to parent(java) and once parent is killed, the child has no idea where to send the output and thus got killed ?

3) I tried to use shutdownhook as explained in this post, but that's not working for me. The code inside the shutdownhook is getting executed only after the java & child processes are done with their work. So, calling process.destroy() inside shutdownhook is not useful here.

This seems obvious, as the JVM is still running in my case, it won't call shutdownhooks until it starts it's shutdown sequence. Don't know how this worked for the other guy, I mean, how come the child process spawned by java is still running when JVM is down.

4) If I restart tomcat, new java process with different pid is generated.

Is it possible to stop the child process programmatically when tomcat is shutdown ?

Let me know if I am not clear with my explanation...

Here is the code that executes system command:

        String command = getCommand();

        File view = new File(viewPath);
        Runtime runtime = Runtime.getRuntime();
        try
        {
            final Process process = runtime.exec(command, null, view);

            StreamReader errorStreamReader = new StreamReader(process
                .getErrorStream());
            Thread errorStreamThread = new Thread(errorStreamReader);
            errorStreamThread.start();
            revisions = parseRevisionLogs(process.getInputStream());

            process.waitFor();

            process.getInputStream().close();
            process.getErrorStream().close();
            process.getOutputStream().close();
        }

Solution

The JVM will not shutdown unless the threads that are left are marked as "daemon". Any non-daemon user threads must finish before the JVM will exit. See this question. If your periodic tasks are not set with setDaemon(true) then they will have to finish before the JVM will exit. You have to call setDaemon before the process starts.

You should be able to make your periodic tasks to be daemon however you do have a race condition with JVM shutdown. You might consider having one daemon task doing the reading from the process but having a non-daemon task do the updating of the index which probably should not get killed while it is working.

Your non-daemon thread could then be sleeping, waiting for the load to finish, and testing to see if it should terminate with a volatile boolean field or other signal.