Search code examples
bashpiping

Efficient way to pipe file contents to program


Assume that we have command cmd that only takes input through a pipe. Given a filename file.txt, what is the most efficient way to pipe this into the command? (I assume cat file.txt | cmd is not very efficient ..)


Solution

  • Let's do a little test with a 1 GB blob (dump.data):

    Using the > operator is much faster than piping from cat:

    $ time cat dump.data | cat >/dev/null
    
    real    0m0.360s
    user    0m0.000s
    sys     0m0.608s
    
    $ time cat <dump.data >/dev/null
    
    real    0m0.158s
    user    0m0.000s
    sys     0m0.156s
    

    The only way that should theoretically be a little faster than < is if cmd accepted a filename as its argument and read the file itself (because there is no IPC involved - only one process works with the data). It does however not make any difference in this test:

    $ time cat dump.data >/dev/null
    
    real    0m0.158s
    user    0m0.000s
    sys     0m0.156s