Search code examples
python-3.xbashshellparallel-processingcron

cron job to execute programs one after the other?


Have six scripts which I want to execute once a day using following logic:
s11, s12 can start and run parallel
s21, s22 should only start after s11 and s12 finished. Both can run parallel
s31, s32 should only start after s21 and s22 finished. Both can run parallel

So far I did it by starting a daily masterscript m by cron. m started all six scripts s11-s32, s11 and s12 did their job directly but the others looked every minute in a counter file and only if the counter had the right value they started the real job. Each script changed the counter before closing, this was the handover to the next script generation. But for other reasons my server was too busy that the new cron started m before the yesterday scripts finished and I screwed up my data.

I assume others had similar problems and know a little library or anything else to get this done properly and stable, for sure the new series shouldn't start before the old finished... .

Thanks in advance for any hints!


Solution

  • A example of master script in bash , i used wait to wait for the completion of
    script that was started in background with &

    This example assume : that all yours scripts are a folder /home/me/myproject/ you have a logs folder where you want to capture some outputs

    #!/bin/bash
    
    cd /home/me/myproject/   
    
    bin/s11 > logs/s11_stdout.log 2>logs/s11_stderr.log   &
    bin/s12 > logs/s12_stdout.log 2>logs/s12_stderr.log   &
    wait
    
    
    ./s21 &
    ./s22 &
    wait
    
    s31 &
    s32 &
    wait