Search code examples
bashperldebianpipinggnu-parallel

GNU parallel not spawning jobs

After an upgrade to Debian 8.6 Jessie the GNU parallel script suddenly stopped parallelizing to more than 2 jobs with the --pipe and -L options.

Before the upgrade the command:

cat file_with_1064_lines.txt | parallel -L10 -j5 -k -v --pipe "wc -l"

spawned 5 processes, which output this:

wc -l
10
wc -l
10
...

The same command after the upgrade:

wc -l
1060
wc -l
4

(The two values above change with respect to the -L option value -- the first is L*floor(1064/L) and the second is 1064 mod L, but there always only two processes outputting.)

The same is observed independently of the parallel version (tested the latest and one from 2013).

PS.

$ uname -a

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux

$ parallel --version

GNU parallel 20161222
Solution

  • -L is the record size. The bug was fixed around 20130122. What you want is to read 1 record of 10 lines:

    parallel -L10 -N1 -j5 -k -v --pipe wc -l

    or 10 records of 1 line:

    parallel -L1 -N10 -j5 -k -v --pipe wc -l