Search code examples
file-ioparallel-processinggnu-parallel

File I/O in gnu parallel


I have a program that takes a single argument. I am using gnu parallel to perform parameter sweeps on this argument. Each run generates a single result, and I want to append all results into a single file, say Results.txt.

What would be a correct way to do this?

I should not have each instance open the file and write to it, as this could create conflicts and also mess up the order of results. The only way I can think of doing this is having each run generate its output in a file with a unique name, and then , when gnu parallel finishes running, merge the results into a single file using a script.

  1. Is there a simpler way of achieving this?
  2. What happens when multiple instances write to/read from the same file? Does gnu parallel create multiple copies, one for each instances, as it does for stdout and stderror?

thanks


Solution

  • If your command sends the result to stdout (standard output) the solution is trivial:

    seq 1000 | parallel echo > Results.txt
    

    GNU Parallel guarantees the output will not be mixed.