Process started with ProcessBuilder stop supplying data

Question

I need to run a unix program from java running on windows. The unix program takes a file name as first parameter, processes the file, and returns the processed file on stdout.

The following works perfectly from CMD console:

wsl.exe /home/user/process /mnt/c/Users/user/input

It prints out the stdout and stderr into console; or I can redirect stdout into a new file which will weigh 1.2MB.

I want to do the same thing from Java. The following code is the most simplified version

ProcessBuilder toCsv = new ProcessBuilder("wsl.exe", "/home/user/process", "/mnt/c/Users/user/input");
Process proc = toCsv.start();
try (InputStream is = proc.getInputStream()){
 for(int i=0; is.read() != -1; i++) {
   System.out.println(i);
 }
}

This always stalls on byte number 380927. Debugger shows that it stalls on FileInputStream.read

• I've tried using redirects to send stdout to a file
• I tried reading only stderr
• I tried reading on a different thread with waitFor on main
• I tried reading on a different thread without mainFor on main
• I tried reading without Thread.sleep instead of waitFor
• I tried passing "/bin/bash", "-c", exec+" "+execFile to ProcessBuilder
• I tried using Runtime.exec instead of ProcessBuilder
• I tried "/bin/bash", "-c", "while :; do echo 1; done". This actually broke the 380k byte limit, and went infinitely, like expected.

What's going on, and how do I fix it?

P.S. the source for the unix executable is here, but I don't think it's very useful, since it works from CMD

Answer 1

I found the answer as I was typing. Apparently, you MUST take care of both STDERR and STDOUT streams somehow, they don't just disappear. If you only read one of them and ignore the other, eventually the ignored one's buffer will get too big and block until it gets emptied.

so adding the following code solved it

new Thread( () -> {
     try (InputStream es = csvBuilder.getErrorStream() ){
            IOUtils.skip(es, Long.MAX_VALUE);
     } catch (IOException e) {
            throw new RuntimeException(e);
     }
}).start();