PowerShell Job throttle blocking the server

Question

I'm experiencing a strange issue with our script server being overloaded and running out of resources. We have a script that copies data from one location to another, this is defined in a large input file that contains over 200 lines of text in the format 'Source path, Destination path'.

We are now in the process of trying to throttle the maximum jobs we kick of at once and I think it's working fine. But for some reason or another we're still running out of resources on the server when the input file contains over 94 lines. This became apparent after some testing.

We tried to upgrade our Windows 2008 R2 server with PowerShell 4.0 to 4 processors and 8 GB of RAM, but no luck. So I assume my throttling isn't working as designed.

Error code:

Insufficient system resources exist to complete the requested service. 

The code:

$MaxThreads = 4
$FunctionFeed = Import-Csv -Path $File -Delimiter ',' -Header 'Source', 'Destination'
$Jobs=@()

Function Wait-MaxRunningJobs {
    Param (
        $Name,
        [Int]$MaxThreads
    )
    Process {
        $Running = @($Name | where State -eq Running)
        while ($Running.Count -ge $MaxThreads) {
            $Finished = Wait-Job -Job $Name -Any
            $Running = @($Name | where State -eq Running)
        }
    }
}

$ScriptBlock = {
    Try {
        Robocopy.exe $Using:Line.Source $Using:Line.Destination $Using:Line.File /MIR /Z /R:3 /W:15 /NP /MT:8 | Out-File $Using:LogFile

        [PSCustomObject]@{
            Source = if ($Using:Line.Source) {$Using:Line.Source} else {'NA'}
            Target = if ($Using:Line.Destination) {$Using:Line.Destination} else {'NA'}
        }
    }
    Catch {
        "Robocopy | ERROR: $($Error[0].Exception.Message)" | 
            Out-File -LiteralPath $Using:LogFile
        throw $($Error[0].Exception.Message)
    }
}

ForEach ($Line in $FunctionFeed) {
    $LogParams = @{
        LogFolder  = $LogFolder
        Name       = $Line.Destination + '.log'
        Date       = 'ScriptStartTime'
        Unique     = $True
    }
    $LogFile = New-LogFileNameHC @LogParams
    ' ' >> $LogFile # Avoid not being able to write to log

    $Jobs += Start-Job -Name RoboCopy -ScriptBlock $ScriptBlock
    Wait-MaxRunningJobs -Name $Jobs -MaxThreads $MaxThreads
}

if ($Jobs) {
    Wait-Job -Job $Jobs
    $JobResults = $Jobs | Receive-Job
}

Am I missing something here? Thank you for your help.

Answer 1

Fixed the problem by enlarging the MaxMemoryPerShellMB for remote sessions from 1GB to 2 GB as described here. Keep in mind that Start-Job is using a remote PowerShell session as mjolinor already indicated, so this variable is applicable to PowerShell jobs.

Solution:

# 'System.OutOfMemoryException error message' when running Robocopy and over 94 PowerShell-Jobs:

Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB # Default 1024
Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048

# Set PowerShell plugins memory from 1 GB to 2 GB
Get-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB # Default 1024
Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB 2048

Restart-Service winrm