Why a batch processing of ffmpeg is freezing the system?

Question

I have a requirement of splitting smaller chunks of videos from 50+ mp4 source files for 5000+ records. Each record may result in 2 or 3 smaller chunks from as many source files out of those 50+.

The logic to determine which source file to be picked up is written in Java and then fed to ffmpeg on Runtime.getRuntime().exec() using ExecutorService with newFixedThreadPool as below:

private static boolean processqueue(ArrayList<String> cmds) {
    final ExecutorService pool;
    int threadsnum = Runtime.getRuntime().availableProcessors()-2;
    pool = Executors.newFixedThreadPool(threadsnum);

    for(final String cmd: cmds){ 
        pool.execute(new Runnable() {
            public void run() {
                System.out.println(cmd);
                try {
                    Runtime.getRuntime().exec(cmd);
                } catch (IOException e) {
                    e.printStackTrace();
                    pool.shutdown();
                }
            }
        });
    }                   
    pool.shutdown();

    // wait for them to finish for up to one minute.
    try {
        if(!pool.awaitTermination(1, TimeUnit.MINUTES)) {
            pool.shutdownNow();
        }

        //Wait a while for tasks to respond to being cancelled
        if(!pool.awaitTermination(1, TimeUnit.MINUTES))
            System.err.println("Pool did not shutdown properly");

    } catch (InterruptedException e) {
        e.printStackTrace();
        pool.shutdownNow();
        //Preserve interrupt status
        Thread.currentThread().interrupt();
        return false;
    }                   

    return true;
}

the String cmd value is one of these based on split or merge requirement:

for split:

ffmpeg -y -ss 00:00:00 -t 00:08 -i E:/tmp/fin12.mp4 -acodec copy -vcodec copy E:/tmp/Intermed/0136f.mp4

or

for merge:

ffmpeg -y -i E:/tmp/Inter/0136c0.mp4 -i E:/tmp/Inter/0136c1.mp4 -i E:/tmp/Inter/0136f.mp4 -i E:/tmp/Jingle.mp4 -i E:/tmp/wm1280.png -filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a][3:v][3:a]concat=n=4:v=1:a=1[vv][a];[vv][4:v]overlay=x=0:y=H-overlay_h[v]" -map "[v]" -map "[a]" E:/tmp/final/0136.mp4

On first attempt, only 250 records were processed. And, on subsequent attempt of balance records processing, it threw below exception; but, processed another 300 records:

java.io.IOException: Cannot run program "ffmpeg": CreateProcess error=1455, The paging file is too small for this operation to complete
at java.lang.ProcessBuilder.start(Unknown Source)

And, this code freezes often. Why is ExecutorService not holding up the queue to process all the records and exit gracefully? What am I doing wrong?

Note : I'm calling Java class from windows batch script by passing relevant arguments which is executed from command line.

Answer 1

You're starting an execution, but not waiting for it to finish, so your threadpool just starts as many processes as there are commands. I'm not sure what the rest of your try/catch is trying to do. I'd suggest you use a CountdownLatch. Here's an example:

public static void main(String[] args) {
    List<String> cmds = Lists.newArrayList("sleep 1", "sleep 2", "sleep 3");

    final ExecutorService pool;
    int threadsnum = Runtime.getRuntime().availableProcessors() - 2;
    pool = Executors.newFixedThreadPool(threadsnum);


    CountDownLatch latch = new CountDownLatch(cmds.size());


    for (final String cmd : cmds) {
        pool.submit(() -> {
            try {
                System.out.println("to be executed: " + cmd);
                Runtime.getRuntime().exec(cmd).waitFor();
                latch.countDown();
            }
            catch (IOException e) {
                Thread.currentThread().interrupt();
            }
            catch (InterruptedException e) {
                e.printStackTrace();
            }
        });

    }

    try {
        latch.await(10, TimeUnit.SECONDS);
        if (latch.getCount() > 0) {
            System.out.println("Waited long enough. There are " + latch.getCount() + " threads still running");
        }
    }
    catch (InterruptedException e) {
        e.printStackTrace();
    }
    finally {
        pool.shutdown();
    }
    System.out.println("Completed.");

}