Search code examples
bashmulticoretiming

bash script only starts new jobs when old batch is finished


I am trying to run 24 versions of the same code on an 8 core machine. The code takes many many hours to run and I only want to run 8 at a time so I was wondering if it was possible to write a bash script which would run 8 and then when those were complete immediately start the next 8 and so on? I basically dont want all 24 to start and then run incredibly slowly! Thanks, Jack

EDIT 1: (More details on the run) The code runs with the following command: nohup ./MyCode MyInputFile 2> Myoutput


Solution

  • You could use gnu parallel

    seq 1 24 | parallel -P 8 ./myscript 
    

    Or with xargs:

    seq 1 24 | xargs -l -P 8 ./myscript 
    

    Update:

    If you want to run the script with Myinput1 Myinput2 Myinput3 .. as parameters you can do

    find . -name 'Myinput*' -print0 | parallel -0 -P 8 ./myscript {1}
    

    or with your command:

    find . -name 'Myinput*' -print0 | parallel -0 -P 8 nohup ./myscript {1}  2> Myoutput