I'm creating a controller script that will run audit scripts contained in separate .ps1 files. I'm using Invoke-Command to kick off jobs on target servers. I know that this will create child jobs for each target computer. How can I timeout individual child jobs if they run too long (ex: 2 minutes)? I know there's Wait-Job -Timeout <seconds>, but that won't really work for my scenario since I want to also print out progress.
Below is what I've tried so far, however the timeout doesn't appear to function.
Any ideas/suggestions? Thanks!
Write-Host "Queueing job... " -NoNewline
Invoke-Command -FilePath $JobFilePath -ComputerName $Targets -AsJob -ThrottleLimit 16 -JobName "AuditJob" -ErrorAction SilentlyContinue | Out-Null
Invoke-Command -ScriptBlock {Get-Job | Wait-job -Timeout 15} -ComputerName $Targets -AsJob -ThrottleLimit 16 -ErrorAction SilentlyContinue | Out-Null
Write-Host "Done" -ForegroundColor Green
$origpos = $host.UI.RawUI.CursorPosition
[Console]::CursorVisible = $false
$Completed = 0
$NotStarted = 0
$Running = 0
$Stopped = 0
$Failed = 0
$Blocked = 0
$Total = 0
$JobsComplete = 0
$CompleteStrLength = 0
do{
$Job = Get-Job -Name "AuditJob" -IncludeChildJob -ErrorAction Stop | Where-Object {$_.Name -ne "AuditJob"}
$Completed = ($Job | Where-Object {$_.State -eq "Completed"}).Count
$NotStarted = ($Job | Where-Object {$_.State -eq "NotStarted"}).Count
$Running = ($Job | Where-Object {$_.State -eq "Running"}).Count
$Stopped = ($Job | Where-Object {$_.State -eq "Stopped"}).Count
$Failed = ($Job | Where-Object {$_.State -eq "Failed"}).Count
$Blocked = ($Job | Where-Object {$_.State -eq "Blocked"}).Count
$Total = $Completed + $NotStarted + $Running +$Stopped + $Failed + $Blocked
$JobsComplete = $Total - $NotStarted - $Running
$PercentComplete = $JobsComplete/$Total
$host.UI.RawUI.CursorPosition = $origpos
Write-Host "[In Progress: $Running || Completed: $Completed || Not Started: $NotStarted || Failed: $($Stopped + $Failed + $Blocked)]"
$CurrentPos = $host.UI.RawUI.CursorPosition
Write-Host (' ' * $CompleteStrLength)
$host.UI.RawUI.CursorPosition = $CurrentPos
$CompleteStr = "$([Math]::Ceiling($PercentComplete * 100))% complete"
$CompleteStrLength = $CompleteStr.Length
Write-Host $CompleteStr
Start-Sleep -Milliseconds 250 #helps with excessive flashing
}while ($JobsComplete -ne $Targets.Count)
So, I think one issue is with your second Invoke-Command. From what I understand the job is being created on your local computer, so that command won't really be doing anything since you are trying to wait for a job that won't exist on the target computer. Using Wait-Job might work locally but getting it to work while your interface is being updated at the same time could be tricky since it pauses the script until the set timeout is reached.
To work around this I created a foreach loop inside your loop to go through the job objects. For each job object, I got start time of the job using the PSBeginTime property and got the elapsed time by subtracting that from the current time using Get-date. If that elapsed is over your set timeout you can use Stop-Job to stop the job.
It does stop the job on the remote machine from continuing and will report on your output as failed. Could require more testing if your remote job is launching an .exe or something as I only tested with a powershell script.
$JobFilePath = ""
$targets = ""
# change as required
$timeOutSeconds = 2
Write-Host "Queueing job... " -NoNewline
Write-Host "Queueing job... " -NoNewline
Invoke-Command -FilePath $JobFilePath -ComputerName $targets -AsJob -ThrottleLimit 16 -JobName "AuditJob" -ErrorAction SilentlyContinue | Out-Null
# remove line
# Invoke-Command -ScriptBlock {Get-Job | Wait-job -Timeout 15} -ComputerName $Targets -AsJob -ThrottleLimit 16 -ErrorAction SilentlyContinue | Out-Null
Write-Host "Done" -ForegroundColor Green
$origpos = $host.UI.RawUI.CursorPosition
[Console]::CursorVisible = $false
$Completed = 0
$NotStarted = 0
$Running = 0
$Stopped = 0
$Failed = 0
$Blocked = 0
$Total = 0
$JobsComplete = 0
$CompleteStrLength = 0
do{
$Job = Get-Job -Name "AuditJob" -IncludeChildJob -ErrorAction Stop | Where-Object {$_.Name -ne "AuditJob"}
$Completed = ($Job | Where-Object {$_.State -eq "Completed"}).Count
$NotStarted = ($Job | Where-Object {$_.State -eq "NotStarted"}).Count
$Running = ($Job | Where-Object {$_.State -eq "Running"}).Count
$Stopped = ($Job | Where-Object {$_.State -eq "Stopped"}).Count
$Failed = ($Job | Where-Object {$_.State -eq "Failed"}).Count
$Blocked = ($Job | Where-Object {$_.State -eq "Blocked"}).Count
foreach ($checkJOb in $Job){
# if job is not started yet time will be null and will error out
if ($null -ne $checkJOb.PSBeginTime){
# calculate elapsed time in seconds.
$totalSecondsElapsed = ((Get-Date) - ($checkJOb.PSBeginTime)).TotalSeconds
# check if time elapsed has passed timeout threshold
if ($totalSecondsElapsed -ge $timeOutSeconds){
# stop the current job
Stop-Job -id $checkJOb.id
}
}
}
$Total = $Completed + $NotStarted + $Running +$Stopped + $Failed + $Blocked
$JobsComplete = $Total - $NotStarted - $Running
$PercentComplete = $JobsComplete/$Total
$host.UI.RawUI.CursorPosition = $origpos
Write-Host "[In Progress: $Running || Completed: $Completed || Not Started: $NotStarted || Failed: $($Stopped + $Failed + $Blocked)]"
$CurrentPos = $host.UI.RawUI.CursorPosition
Write-Host (' ' * $CompleteStrLength)
$host.UI.RawUI.CursorPosition = $CurrentPos
$CompleteStr = "$([Math]::Ceiling($PercentComplete * 100))% complete"
$CompleteStrLength = $CompleteStr.Length
Write-Host $CompleteStr
Start-Sleep -Milliseconds 250 #helps with excessive flashing
}while ($JobsComplete -ne $Targets.Count)