Restart the process if its output unchanged in PowerShell

I have this command

for ($i=0; $i -gt -1; $i++) {
    $path = "..."
    ffmpeg -f dshow -i audio="Microphone (USB Microphone)" -y -t 00:10:00 -b:a 128k $path
}

I need to get the current state of the last line of the command output stream, then if the line remains unchanged (staled/freezing/hanging) over 5 seconds log "warning: the program is freezing. trying to restart...", then stop the process and re-start the command.

But I wonder, is it even possible? Thanks for your help.

Solution

You can use a job to run a program in the background, which enables you to monitor its output and check for timeout conditions.

Note:

The code below uses the Start-ThreadJob cmdlet, which offers a lightweight, much faster thread-based alternative to the child-process-based regular background jobs created with Start-Job.
- Start-ThreadJob comes with PowerShell (Core) 7+ and in Windows PowerShell can be installed on demand with, e.g., Install-Module ThreadJob -Scope CurrentUser.
- In most cases, thread jobs are the better choice, both for performance and type fidelity - see the bottom section of this answer for why.
Start-ThreadJob is compatible with the other job-management cmdlets, such as the Receive-Job cmdlet used below. If you're running Windows PowerShell and installing the module is not an option, simply replace Start-ThreadJob with Start-Job in the code below.
The code uses a simplified ffmpeg call (which should have no impact on functionality), and can be run as-is, if you place a sample.mov file in the current directory, which will transcode it to a sample.mp4 file there.
Due to applying redirection 2>&1 to the ffmpeg call, it is the combined stdout and stderr that is monitored, so that status and progress information, which ffmpeg emits to stderr, is also monitored.
- So as to emit progress messages in place (on the same line), as direct invocation of ffmpeg does, a CR ("`r") is prepended to each and [Console]::Error.Write() rather than [Console]::Error.WriteLine() is used.
- Caveat: Writing to [Console]::Error bypasses PowerShell's system of output streams. This means that you'll neither be able to capture nor silence this output from inside a PowerShell session in which your script runs. However, you can capture it - via a 2> redirection - using a call to the PowerShell CLI, e.g.:
```
 powershell.exe -File C:\path\to\recording.ps1 2>stderr_output.txt
```
  - Note that each and every progress message is invariably captured separately, on its own line.
- What constitutes a progress message is inferred from each line's content, using regex '^\w+= ', so as to match progress lines such as frame= … and size= …

while ($true) { # Loop indefinitely, until the ffmpeg call completes.

  Write-Verbose -Verbose "Starting ffmpeg..."

  # Specify the input file path.
  # Output file will be the same, except with extension .mp4
  $path = './sample.mov'
  
  # Use a job to start ffmpeg in the background, which enables
  # monitoring its output here in the foreground thread.
  $jb = Start-ThreadJob { 
    # Note the need to refer to the caller's $path variable value with $using:path
    # and the 2>&1 redirection at the end.
    # Once the output file exists, you can simulate freezing by removing -y from the call:
    # ffmpeg will get stuck at an invisible confirmation prompt.
    ffmpeg -y -i $using:path ([IO.Path]::ChangeExtension($using:path, '.mp4')) 2>&1
    $LASTEXITCODE # Output ffmpeg's exit code.
  }
  
  # Start monitoring the job's output.
  $sw = [System.Diagnostics.Stopwatch]::StartNew()
  do {
    Start-Sleep -Milliseconds 500 # Sleep between attempts to check for output.
    # Check for new lines and pass them through.
    $linesReceived = $null
    $jb | Receive-Job -OutVariable linesReceived | 
      ForEach-Object { 
        if ($_ -is [int]) {  
          # The value of $LASTEXITCODE output by the job after ffmpeg exited.
          $exitCode = $_
        }
        elseif ($_ -is [string]) { 
          # Stdout output: relay as-is
          $_
        } 
        else {
          # Stderr output, relay directly to stderr (bypassing PowerShell's error stream).
          # If it looks like a *progress* message, print it *in place*
          # Note: If desired, the blinking cursor could be (temporarily) turned off with [Console]::CursorVisible = $false
          if (($line = $_.ToString()) -match '^\w+= ') { [Console]::Error.Write("`r$line") } 
          else                                         { [Console]::Error.WriteLine($line) } 
        } 
      }
    if ($linesReceived) { $sw.Restart() } # Reset the stopwatch, if output was received.
  } while (($stillRunning = $jb.State -eq 'Running') -and $sw.Elapsed.TotalSeconds -lt 5)

  # Clean up the job forcefully, which also aborts ffmpeg, if it's still running.
  $jb | Remove-Job -Force

  # Handle the case where ffmpeg exited.
  # This can mean successful completion or an error condition (such as a syntax error)
  if (-not $stillRunning) {
    # Report the exit code, which implies success (0) vs. failure (nonzero).
    Write-Verbose -Verbose "The program exited without freezing, with exit code $exitCode."
    break # Exit the loop.
  }

  # ffmpeg froze: issue a warning and restart (continue the loop).
  Write-Warning "The program is freezing. Restarting..."  

}

# Exit with the exit code relayed from ffmpeg.
exit $exitCode