Process.waitFor hangs even though stdout and stderr get read

Question

I already searched the web, and the question process.waitFor() never returns indicates that it is often a problem with processes that their stdout or stderr do not get read.

We use ProcessBuilder with redirectOutput and redirectError to achieve this, and I think we should be on the safe side, see the following method we use to execute processes:

public static void execute(String directory, long timeout, File out, File err, String... command) throws InterruptedException, IOException {
    LOGGER.log(Level.INFO, String.format("executing command %s (%s)", Arrays.toString(command), timeout > 0 ? String.format("timeout = %,d[ms]", timeout) : "no timeout"));
    ProcessBuilder builder = new ProcessBuilder();
    builder.directory(new File(directory));
    builder.command(command);
    builder.redirectOutput(out);
    if(out == err) {
        builder.redirectErrorStream(true);          
    } else {
        builder.redirectError(err);
    }
    long time = System.currentTimeMillis();
    Process process = builder.start();
    try {
        LOGGER.log(Level.FINE, "waiting for process");
        boolean exited = process.waitFor(timeout, TimeUnit.MILLISECONDS);
        if(!exited) {
            LOGGER.log(Level.WARNING, "timeout reached, trying to destroy ...");
            exited = destroy(silent, process); // Helper method to destroy processes
        }
        long duration = System.currentTimeMillis() - time;
        int exitValue = process.exitValue();
        LOGGER.log(Level.INFO, "execution finished in " + duration + "[ms] => " + exitValue);
    } catch (InterruptedException | Error | RuntimeException e) {
        LOGGER.log(Level.SEVERE, "execution failed", e);
        throw e;
    }
}

Yet, the problem is that it hangs on the process.waitFor(timeout, TimeUnit.MILLISECONDS) call, even though the process should easily have finished within the timeout.

The logging output is

Oct 07, 2017 12:39:55 AM at.ProcessExecutor execute
INFO: executing command [java, -Xmx8G, -XX:+UseG1GC, -XX:+CrashOnOutOfMemoryError, -jar, MetricCalc.jar] (timeout = 14,400,000[ms])
Oct 07, 2017 12:39:55 AM at.ProcessExecutor execute
FINE: waiting for process

(recognize that no execution finished line is written yet)

The err file reads

... Things we write to std.err ...
Finished Metrics

and the main method of MetricCalc looks like

public static void main(String[] args) {
    .... do some stuff ...
    System.err.println("Finished Metrics");
}

which indicates that the reading works fine, the last line of the Java program has been executed and the process should have terminated.

Anyone having an idea why the process does not terminate / it still hangs on Process.waitFor()?

Answer 1

It was not a problem with Process.waitFor() but a problem with process termination.

The java application that got started used an ExecutorService that did not got shotdown correctly and left zombie threads alive which prevented process termination.

Adding executorService.shutdown() solved the problem, and the application now terminates as expected.