Powershell Pipeline data to external console application

Question

I have a console application which can take standard input. It buffers up the data until the execute command, at which point it executes it all, and sends the output to standard output.

At the moment, I am running this application from Powershell, piping commands into it, and then parsing the output. The data piped in is relatively small; however this application is being called about 1000 times. Each time it is executed, it has to load, and create network connections. I am wondering whether it might be more efficient to pipeline all the commands into a single instantiation of the console application.

I have tried this by adding all Powershell script, that manufactures the standard input for the console, into a function, then piping that function to the console application. This seems to work at first, but you eventually realise it is buffering up all the data in Powershell until the function has finished, then sending it to the console's StdIn. You can see this because I have a whole load of Write-Host statements that flash by, and only then do you see the output.

e.g.

Function Run-Command1
{
    Write-Host "Run-Command1"
    "GET nethost xxxx COLS id,name"
    "EXEC"
}

Function Run-Command2
{
    Write-Host "Run-Command2"
    "GET nethost yyyy COLS id,name"
    "GET users yyyy COLS id,name"
    "EXEC"
}

...

Function Run-CommandX 
{
...
}

Previously, I would use this as:

Run-Command1 | netapp.exe -connect QQQQ -U user -P password
Run-Command2 | netapp.exe -connect QQQQ -U user -P password
...
Run-CommandX | netapp.exe -connect QQQQ -U user -P password

But now I would like to do:

Function Run-Commands
{
    Run-Command1
    Run-Command2
    ...
    Run-CommandX
}

Run-Commands |
netapp.exe -connect QQQQ -U user -P password

Ideally, I would like the Powershell pipeline behaviour to be extended to an external application. Is this possible?

Answer 1

EDIT: As @mklement0 pointed out, this is different in PowerShell Core.

In PowerShell 5.1 (and lower) think you would have to manually write each pipeline item to the external application's input stream.

Here's an attempt to build a function for that:

function Invoke-Pipeline {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory, Position = 0)]
        [string]$FileName,

        [Parameter(Position = 1)]
        [string[]]$ArgumentList,

        [int]$TimeoutMilliseconds = -1,

        [Parameter(ValueFromPipeline)]
        $InputObject
    )
    begin {
        $process = [System.Diagnostics.Process]::Start((New-Object System.Diagnostics.ProcessStartInfo -Property @{
            FileName = $FileName
            Arguments = $ArgumentList
            UseShellExecute = $false
            RedirectStandardInput = $true
            RedirectStandardOutput = $true
        }))
        $output = [System.Collections.Concurrent.ConcurrentQueue[string]]::new()
        $event = Register-ObjectEvent -InputObject $process -EventName 'OutputDataReceived' ` -Action {
             $Event.MessageData.TryAdd($EventArgs.Data)
        } -MessageData $output
        $process.BeginOutputReadLine()
    }
    process {
        $process.StandardInput.WriteLine($InputObject)
        [string]$line = ""
        while (-not ($output.TryDequeue([ref]$line))) {
            start-sleep -Milliseconds 1
        }
        do {
            $line
        } while ($output.TryDequeue([ref]$line))
    }
    end {
        if ($TimeoutMilliseconds -lt 0) {
            $exited = $process.WaitForExit()
        }
        else {
            $exited = $process.WaitForExit($TimeoutMilliseconds)
        }
        if ($exited) {
            $process.Close()
        }
        else {
            try {$process.Kill()} catch {}
        }
    }
}

Run-Commands | Invoke-Pipeline netapp.exe "-connect QQQQ -U user -P password"

The problem is, that there is no perfect solution, because by definition, you cannot know when the external program will write something to its output stream, or how much.

Note: This function doesn't redirect the error stream. The approach would be the same though.