Search code examples
powershellparallel-processingjobs

Multiple jobs in one Powershell code with different duration


I have a Powershell code - with several scriptblocks. Every script block is a job, and I start the jobs in a while($true) section. The problem is that the job duration is different, one job runs ~1minutes, other ones 7minutes. I want to run the jobs continuously, and of course, I want to wait for the job ends before I start the new one.
I would like to not wait for the end of the longest job, but start a new one of a shorter job. For example, if the "job1" is finishing in 1 minute, but the "job2" takes 10 minutes, I would like to start 10 "job1" during the "job2" process.

Do you have an idea of how I can reach this?

while($true) {

   Start-job -name somejob1 -scriptblock $block1...

   Start-job -name somejob2 -scriptblock $block2...

   ...

   Start-job -name somejobX -scriptblock $blockX..


   Get-job | Wait-job | Remove job

   Start-sleep -s 120

}

Solution

  • Using your current code, one possibility is to only a repeat a job once the previous one completes with a few if statements.

    while($true) {
    
    if ((Get-Job).Name -notcontains "startjob1") {
        Start-Job -Name "startjob1" -scriptblock $block1
    }
    
    if ((Get-Job).Name -notcontains "startjob2") {
        Start-Job -Name "startjob2" -scriptblock $block2
    }
    
    if ((Get-Job).Name -notcontains "startjob3") {
        Start-Job -Name "startjob3" -scriptblock $block3
    }
    
    Start-Sleep -Seconds 120
    
    Get-Job -State "Completed" | Remove-Job
    
    }
    

    Explanation:

    This will produce an infinite loop of starting jobs that are not listed in the Get-Job output. After each job is started and has its state evaluated, we wait 120 seconds before checking the job statuses. At this point, jobs that completed successfully are removed. This part can be changed to check against other status values using | Where-Object before piping to Remove-Job since there may be times when jobs fail and you want to rerun anyway.

    In general, you can utilize Wait-Job to wait for a job to complete or for a timeout period, whichever comes first. If you know more about which jobs run the shortest or you know of a more suitable timeout period, then modify the timeout value or reorder the Start-Job commands.

    Start-job -name somejob1 -scriptblock $block1...
    Wait-Job -name somejob1 -timeout 60
    
    Start-job -name somejob2 -scriptblock $block1...
    Wait-Job -name somejob2 -timeout 60