Search code examples
bashparallel-processinggnu-parallel

GNU Parallel: how to prevent specific jobs from being processed in parallel

GIVEN:

A set of jobs to be run in parallel: { app0, app1, app2, .... }

QUESTION:

How is it possible to initiate the tool 'GNU parallel' to run all jobs in parallel, whereby some specific jobs prevented from running concurrently?

EXAMPLE:

If appX and appY rely on the same resources, how can one specify that appX may run in parallel with app0, app1, ... but never with appY?

EXAMPLE 2:

appX and appY may run in parallel, but neither of them shall be running concurrently with appZ.

Solution

  • It is not 100% clear to me what you want. Maybe replace appX and appY with:

    sem --id myidXY --fg appX
sem --id myidXY --fg appY

    Which can be done like this:

    ... | parallel eval '{= s/(app(X|Y))/sem --id appXY --fg $1/ =}'

    This should make sure only a single appX or appY is running; but let plenty of appZs running.

    {= =} is interpreted as Perl code.

    s/(app(X|Y))/sem --id appXY --fg $1/ replaces appX or appY with sem --id appXY --fg followed by either appX or appY depending in what was matched. If nothing is matched then the value is unchanged.

    (echo appX; echo appX; echo appX; 
 echo appY; echo appX; echo appV;
 echo appX; echo appZ) |
  parallel eval '{= s/(app(X|Y))/sem --id appXY $1/ =}'

    If that is not what you mean, please edit the question.