I have optimized my code for performance by utilizing ForEach-Object -AsJob -Parallel ....
However, it comes with major drawbacks, as I can't write debug messages to the console, nor debug my code by stepping through it.
So, every time I want to debug my code, I have to remove -AsJob and -Parallel options, and then e.g. add the debug logging statements, which is something I want to avoid(!). I don't see how this scale - what if I had hundreds of parallel for each loops that all appeared like "black boxes".
How can I write debug messages to the console and step through my code, when using ForEach-Object -AsJob -Parallel ...?
$appPermissionsJob = $servicePrincipals | ForEach-Object -AsJob -ThrottleLimit $ThrottleLimit -Parallel {
$spApplicationPermissions = $using:spApplicationPermissions
$servicePrincipalId = $_.Id
try {
$applicationPermissions = Get-MgServicePrincipalAppRoleAssignedTo -ServicePrincipalId $servicePrincipalId -All -PageSize 999
if ($applicationPermissions -ne $null) {
$applicationPermissions | ForEach-Object {
Write-Host "Processing service principal ${servicePrincipalId} with app role $($_.ResourceId)"
if ($_.ResourceId -eq $using:MSGraphServicePrincipalId) {
$item = New-Object PSObject -Property ([ordered] @{
ServicePrincipalId = $_
ApplicationPermissions = $applicationPermissions
})
$spApplicationPermissions.TryAdd($servicePrincipalId, $item)
}
}
}
}
catch {
Write-Verbose "Failed to download delegated permissions for service principal ${servicePrincipalId}: $($_.Exception.Message)"
$dictFailed.TryAdd($servicePrincipalId, $_.Exception.Message)
}
}
Generally, cmdlets that accept script blocks expect those script blocks to control their stream output themselves, with the caller needing to use redirections in order to capture or suppress such output.
As such, unless output of a given stream, such as the verbose stream in the case at hand, is turned on inside the script block, it won't surface in the caller's context.
Therefore, in order to surface verbose-stream output, you must either use Write-Verbose -Verbose ... calls inside your script block (i.e. using the common -Verbose parameter), or set $VerbosePreference = 'Continue' at the start of the block.
However, due to a putative bug, as of PowerShell (Core) 7.4.x, output from thread-based parallelism - both via ForEach-Object -Parallel (with or without -AsJob) and Start-ThreadJob - is unexpectedly additionally filtered by the caller's stream-controlling settings (whether via common parameters or the equivalent preference variables).
Here are minimal examples that provide a workaround:
# !! The *outer* -Verbose arguably shouldn't be necessary in either case,
# !! but is, as of PowerShell 7.4.x
ForEach-Object -Parallel { Write-Verbose hi -Verbose } -Verbose
ForEach-Object -AsJob -Parallel { Write-Verbose hi -Verbose } |
Receive-Job -Wait -AutoRemoveJob -Verbose
ForEach-Object -AsJob -Parallel script block:The short of it is that debugging such jobs, as well as the technologically closely related thread jobs created with Start-ThreadJob, via the Debug-Job cmdlet, is currently (as of PowerShell 7.4.x) very limited and cumbersome, due to numerous bugs.
A conceptual challenge is that copies of the code in the script block passed to -Parallel run in multiple runspaces (threads) by design, one for each pipeline input object, albeit capped by the so-called throttle limit, i.e. the max. number of runspaces that are allowed to run simultaneously at a given time; the default limit is 5, but can be modified via the -ThrottleLimit parameter.
Therefore, you can only debug one of these runspaces at a time.
While placing a Wait-Debugger statement at the start of the script block should in theory allow you to target each runspace sequentially, a bug currently prevents the use of Wait-Debugger:
The upshot is that you currently cannot make runspaces wait for a debugger to attach.
Even a workaround via an aux. function and a Set-PSBreakpoint -Command call from inside the script block is currently not an option, due to a separate bug:
See GitHub issue #39.
Debug-Job fails, unless the job has already entered the Running state after creation.
Not only do you therefore need to ensure that the job is in the desired state manually, the Running state is reported prematurely, requiring an artificial sleep interval to work around this bug:
Generally, it would be nice to be able to start a job in a manner that makes the parallel runspaces hold off on executing until a debugger connects, or to be able to set breakpoints from outside the job's script block.
-Breakpoints and -WaitDebugger parameters to the dedicated job cmdlets (Start-Job and Start-ThreadJob), which would have to be added to the ForEach-Object -AsJob -Parallel parameter set too.Here's a minimal example that shows what you can currently do (as of PowerShell 7.4.x):
# Start a sample job with 3 parallel runspaces.
$job = 1..3 | ForEach-Object -AsJob -Parallel {
# Dummy activity that last about two seconds, so that the
# debugger has a chance to attach.
# Note that using `Wait-Debugger` does NOT currently work.
$rs = [runspace]::DefaultRunspace
1..20 | ForEach-Object {
$i = $_
"Hi #$i from runspace $($rs.Name)"
Start-Sleep -Milliseconds 100
}
}
# Wait until the job is ready to be debugged.
while ($job.State -ne 'Running') { Start-Sleep -Milliseconds 50 }
# !! Due to a bug, 'Running' is reported *too soon*.
# !! The suboptimal workaround is to sleep a little.
Start-Sleep -Milliseconds 100
# Invoke the debugger:
# Debug-Job can only debug one runspace at a time.
# Each runspace runs in a *child* job. Here, the first one
# is targeted.
# !! It is non-deterministic at which statement the debugger will
# !! break.
# Submit 'd' or 'c' to exit the debugging session and continue running,
# 'h' to display help.
$job.ChildJobs[0] | Debug-Job
# After exiting the debugger,
# receive and output the job's output, then delete the job.
$job | Receive-Job -Wait -AutoRemoveJob