Search code examples
bashparallel-processinggnu-parallel

GNU Parallel: Giving a hint about execution time

GIVEN:

Some very much time-consuming jobs, BIG0, BIG1, etc. and a bunch of small fast jobs fast0, fast1, etc.

QUESTION:

Assume, that the BIG* jobs take 2 hours, and the fast* jobs take 1 hour. How can I instruct GNU Parallel to distribute the jobs over time and cores as follows

   time[h]    CPU[0]  CPU[1]     
     0        BIG0    BIG1     
     1         :        :       
     2        fast0   fast1

and not as shown below?

   time[h]   CPU[0]   CPU[1] 
     0       BIG0     fast0   
     1         :      fast1
     2       BIG1            
     3         :

The last setup would take one hour longer.

Solution

  • One possible approach might be to sort your jobs by expected time, before submitting to GNU Parallel, big ones first:

    parallel ... ::: BIG* fast*