I actually WANT a blocking write to a linux named pipe

Question

I have a threaded application. In one thread I want to write data streamed from a database to a named pipe and I want the writes to block if the reader (the command line program "zip" in this case) can not keep up with the java thread. Data that goes into a single Zip file entry might be bigger than main memory of the system.

I see the following behaviour regardless if using FileOutputStream or FileWriter when writing to the named pipe: The writes would buffer until the java heap is filled up and then actually slow down the thread to the reader's speed. For a single threaded process this is only a waste of space but for a multithreaded process this lets run other threads into out of memory exceptions.

The only remaining option I see is to use JNA to do a blocking write in C. Other suggestions are welcome.

BTW. the very reason that I let the "zip" tool do the compression is that java.util.zip and Lingala's Zip4J would fill the RAM with buffers.

OK here is a small condensed example. I create a named pipe "fifo" using "mkfifo fifo" and start "zip --fifo -fz -v fifo.zip fifo" to have the reading process block on the named pipe. Then I start the follwoing java program with say -Xmx32M. Without the "Mem Eater" thread it behaves as described above. With it this thread will run in an OutOfMemoryException. Now for the code:

import java.io.FileWriter;
import java.util.LinkedList;
import java.util.List;

public class fos {

    public static void main(String[] argv) {
        if (argv.length != 1) {
            System.err.println("Usage is:");
            System.err.println("java fos.java <fifo>");
            System.exit(-1);
        }

        String fifoName = argv[0];
        
        startMemConsumerThread();

        try (var fifoWriter = new FileWriter(fifoName)) {
            for(long i=0L; i< Long.MAX_VALUE; i++)
                fifoWriter.write("Hello World! "+i+"\r\n");
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

    private static void startMemConsumerThread() {
        final int NUM_CHUNKS = 20;
        final int CHUNK_SIZE = 1024*1024;
        final List< byte[] > chunks = new LinkedList<>();
        
        var t = new Thread("Mem Eater") {
            @Override
            public void run() {
                while (true) {
                    while (chunks.size() < NUM_CHUNKS)
                        chunks.add(new byte[CHUNK_SIZE]);
                    chunks.remove(NUM_CHUNKS % 7);
                }
            }
        };
        
        t.setDaemon(true);
        t.start();
    }
}

Answer 1

I did a thin JNA wrapper around open, write, close to have the blocking behavior of write. Now I no longer observe unbounded memory consumption but the Java heap will grow to the configured maximum and GC will handle it gracefully.

I have to correct myself: even when using FileWriter.write the blocking takes place and the memory consumption is bound. Thanks to joni for pushing me!

But things go out of control when using java.util.ZipOutputStream for this use case. But that is another story...