Search code examples
arraysmultithreadingpowershellimagemagick-convertstart-job

How to put powershell commands in an array?


I have a script that does a bunch of image processing. So far it runs sequentially, uses only one core and takes forever. I have four cores at hand. I want to run four of these commands at the same time. To make this happen, I need to put the commands in an array. These are the original commands:

convert.exe -density 200 -quality 80 -delete 0 -scene 1  C:\Users\mles\Desktop\ta2014\v33_1_21_Northland.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 22 C:\Users\mles\Desktop\ta2014\v33_22_31_Auckland.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 32 C:\Users\mles\Desktop\ta2014\v33_32_49_Waikato.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 50 C:\Users\mles\Desktop\ta2014\v33_50_62_Whanganui.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 63 C:\Users\mles\Desktop\ta2014\v33_63_69_Manawatu.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 70 C:\Users\mles\Desktop\ta2014\v33_70_75_Wellington.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 76 C:\Users\mles\Desktop\ta2014\v33_76_92_Marlborough.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 93 C:\Users\mles\Desktop\ta2014\v33_93_117_Canterbury.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 118 C:\Users\mles\Desktop\ta2014\v33_118_127_Otago.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg
convert.exe -density 200 -quality 80 -delete 0 -scene 128 C:\Users\mles\Desktop\ta2014\v33_128_141_Southland.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg

So far I came up with this:

$array = @()
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 1  C:\Users\mles\Desktop\ta2014\v33_1_21_Northland.pdf C:\Users\mles\Desktop\ta2014\%03d_test.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 22 C:\Users\mles\Desktop\ta2014\v33_22_31_Auckland.pdf C:\Users\mles\Desktop\ta2014\%03d_test.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 32 C:\Users\mles\Desktop\ta2014\v33_32_49_Waikato.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 50 C:\Users\mles\Desktop\ta2014\v33_50_62_Whanganui.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 63 C:\Users\mles\Desktop\ta2014\v33_63_69_Manawatu.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 70 C:\Users\mles\Desktop\ta2014\v33_70_75_Wellington.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 76 C:\Users\mles\Desktop\ta2014\v33_76_92_Marlborough.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 93 C:\Users\mles\Desktop\ta2014\v33_93_117_Canterbury.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 118 C:\Users\mles\Desktop\ta2014\v33_118_127_Otago.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"
$array += "convert.exe -density 200 -quality 80 -delete 0 -scene 128 C:\Users\mles\Desktop\ta2014\v33_128_141_Southland.pdf C:\Users\mles\Desktop\ta2014\%03d.jpg"

foreach ($element in $array) {
   $MaxThreads = 4
   While (@(Get-Job | Where { $_.State -eq "Running" }).Count -ge $MaxThreads)
   {  Write-Host "Waiting for open thread...($MaxThreads Maximum)"
      Start-Sleep -Seconds 1
   }

   Start-Job -Scriptblock{ $element }
}

While (@(Get-Job | Where { $_.State -eq "Running" }).Count -ne 0)
{  Write-Host "Waiting for background jobs..."
   Get-Job    #Just showing all the jobs
   Start-Sleep -Seconds 1
}

ForEach ($Job in (Get-Job)) {
   Remove-Job $Job
}

The jobs start but quit immediately. I guess it has to do with how I'm storing the commands in the array?


Solution

  • You may want to take a look at workflows but they bump up against a max of 5 concurrent processes. There are also several examples of using runspace pools and powershell to implement multi threading in a manner that bypasses this limit. I gave a working example here.

    Alternatively just store the scriptblocks as scriptblocks in the array instead of as strings.

    $commands = @()
    $commands += {echo "test1"}
    $commands += {echo "test2"}
    $commands += {echo "test3"}
    foreach($command in $commands){
        start-job $command
    }