Running multiple scriptblocks at the same time with Start-Job (instead of looping)

Hi all!

I've been looking for a way to make my script more efficient and I've come to the conclusion (with help from the nice people here on StackOverflow) that Start-Job is the way to go.

I have the following foreach-loop that I would like to run simultanously on all the servers in $servers. I have problems understanding how I actually collect the information returned from Receive-Job and add to $serverlist.

PS: I know that I am far away from getting this nailed down, but I would really appreciate some help starting out as I am quite stumped on how Start-Job and Receive-Job works..

# List 4 servers (for testing)
$servers = Get-QADComputer -sizelimit 4 -WarningAction SilentlyContinue -OSName *server*,*hyper*

# Create list
$serverlistlist = @()

# Loop servers
foreach($server in $servers) {

    # Fetch IP
    $ipaddress = [System.Net.Dns]::GetHostAddresses($Server.name)| select-object IPAddressToString -expandproperty IPAddressToString

    # Gather OSName through WMI
    $OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $server.name ).caption

    # Ping the server
    if (Test-Connection -ComputerName $server.name -count 1 -Quiet ) {
        $reachable = "Yes"
    }

    # Save info about server
    $serverInfo = New-Object -TypeName PSObject -Property @{
        SystemName = ($server.name).ToLower()
        IPAddress = $IPAddress
        OSName = $OSName
    }
    $serverlist += $serverinfo | Select-Object SystemName,IPAddress,OSName
}

Notes

I am outputting $serverlist to a csv-file at the end of the script
I list aprox 500 servers in my full script

Solution

Since your loop only needs to work with a string it's easy to turn it into a concurrent script.

Below is an example of making making your loop use background jobs to speed up processing.

The code will loop through the array and spin up background jobs to run the code in the script block $sb. The $maxJobs variable controls how many jobs run at once and the $chunkSize variable controls how many servers each background job will process.

Add the rest of your processing in the script block adding whatever other properties you want to return to the PsObject.

$sb = {
    $serverInfos = @()
    $args | % {
        $IPAddress = [Net.Dns]::GetHostAddresses($_) | select -expand IPAddressToString
        # More processing here... 
        $serverInfos += New-Object -TypeName PsObject -Property @{ IPAddress = $IPAddress }
    }
    return $serverInfos
}

[string[]] $servers = Get-QADComputer -sizelimit 500 -WarningAction SilentlyContinue -OSName *server*,*hyper* | Select -Expand Name

$maxJobs = 10 # Max concurrent running jobs.
$chunkSize = 5 # Number of servers to process in a job.
$jobs = @()

# Process server list.
for ($i = 0 ; $i -le $servers.Count ; $i+=($chunkSize)) {
    if ($servers.Count - $i -le $chunkSize) 
        { $c = $servers.Count - $i } else { $c = $chunkSize }
    $c-- # Array is 0 indexed.

    # Spin up job.
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList ( $servers[($i)..($i+$c)] ) 
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $maxJobs) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

$jobs | Receive-Job | Select IPAddress

Here is the version that processes a single server per job:

$servers = Get-QADComputer -WarningAction SilentlyContinue -OSName *server*,*hyper*

# Create list
$serverlist = @()

$sb = {
    param ([string] $ServerName)
    try {
        # Fetch IP
        $ipaddress = [System.Net.Dns]::GetHostAddresses($ServerName)| select-object IPAddressToString -expandproperty IPAddressToString

        # Gather OSName through WMI
        $OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $ServerName ).caption

        # Ping the server
        if (Test-Connection -ComputerName $ServerName -count 1 -Quiet ) {
            $reachable = "Yes"
        }

        # Save info about server
        $serverInfo = New-Object -TypeName PSObject -Property @{
            SystemName = ($ServerName).ToLower()
            IPAddress = $IPAddress
            OSName = $OSName
        }
        return $serverInfo
    } catch {
        throw 'Failed to process server named {0}. The error was "{1}".' -f $ServerName, $_
    }
}

# Loop servers
$max = 5
$jobs = @()
foreach($server in $servers) {
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList $server.Name
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $max) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

# Check for failed jobs.
$failed = @($jobs | ? {$_.State -eq 'Failed'})
if ($failed.Count -gt 0) {
    $failed | % {
        $_.ChildJobs[0].JobStateInfo.Reason.Message
    }
}

# Collect job data.
$jobs | % {
    $serverlist += $_ | Receive-Job | Select-Object SystemName,IPAddress,OSName
}