Search code examples
gnu-parallel

GNU Parallel jobs in background


I have questions when using gnu parallel to run my jobs. Despite setting --results argument and end my command with an "&", bash still show whatever outputs from my command to the screen. Here is a general structure of my codes (assuming the program I run is called "command", and filenames.txt store line-by-line all the input files):

parallel -a filenames.txt -j 6 --results logs/ --joblog joblog command --input {} --output {}.out &

In addition, I wonder if using disown in this situation would cause any problems. In one of my run, it is weird that the joblog file appears to be interrupted and stop updating despite more jobs had been run. From the output files (stderr, set by --results), most of the jobs that were not recorded by joblog file were actually completed, with some incomplete stderr. I wasn't sure if those jobs with incomplete stderr completed successfully or not because the output files (set by --output) looks normal in file size. I checked the timestamp of the joblog file, the last update time may have been when I used disown (although I am not certain).

In general, from the manual and tutorial of GNU Parallel, I am not sure the compatibility of running parallel in background.

Thanks!

J


Solution

  • --results also prints a copy to stdout/stderr. This is likely to change in the future (see https://lists.gnu.org/archive/html/parallel/2020-12/msg00003.html).

    Right now you simply redirect the output to /dev/null.

    I have long ago stopped using disown. Instead I log in to the remote server, start tmux, and run jobs in tmux. disown should not affect GNU Parallel, though, so I cannot explain the weird situation you see with --joblog.