Streaming commands output progress from Goroutine

Question

Streaming commands output progress question addresses the problem of printing progress of a long running command.

I tried to put the printing code within a goroutine but the scanner claims to have already hit the EOF immediately and the for block is never executed.

The bufio.scan code that gets executed on the first execution of the Scan() method is:

    // We cannot generate a token with what we are holding.
    // If we've already hit EOF or an I/O error, we are done.
    if s.err != nil {
        // Shut it down.
        s.start = 0
        s.end = 0
        return false
    }

And if I print s.err the output is EOF.

The code I'm trying to run is:

cmd := exec.Command("some", "command")
c := make(chan int, 1)

go func(cmd *exec.Cmd, c chan int) {
    stdout, _ := cmd.StdoutPipe()

    <-c

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
}(cmd, c)

cmd.Start()

c <- 1

cmd.Wait()

The idea is to start the Goroutine, get a hold of the cmd.stdout, wait that the cmd is started, and start processing its output.

The result is that the long command gets executed and the program waits for its completion, but nothing is printed to terminal.

Any idea why by the time scanner.Scan() is invoked for the first time the stdout has already reached EOF?

Answer 1

There are some problems:

• The pipe is being closed before reading all data.
• Always check for errors
• Start cmd.Start() after c <- struct{}{} and use unbuffered channel c := make(chan struct{})

---

Two working sample codes:

1: Wait using channel then close the pipe after EOF using defer func() { c <- struct{}{} }(), like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
)

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})

    go run(cmd, c)

    c <- struct{}{}
    cmd.Start()

    <-c
    if err := cmd.Wait(); err != nil {
        fmt.Println(err)
    }
    fmt.Println("done.")
}

func run(cmd *exec.Cmd, c chan struct{}) {
    defer func() { c <- struct{}{} }()
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        panic(err)
    }
    <-c
    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
    }
    fmt.Println("EOF")
}

---

2: Also you may Wait using sync.WaitGroup, like this working sample code:

package main

import (
    "bufio"
    "fmt"
    "os/exec"
    "sync"
)

var wg sync.WaitGroup

func main() {
    cmd := exec.Command("Streamer")
    c := make(chan struct{})
    wg.Add(1)
    go func(cmd *exec.Cmd, c chan struct{}) {
        defer wg.Done()
        stdout, err := cmd.StdoutPipe()
        if err != nil {
            panic(err)
        }
        <-c
        scanner := bufio.NewScanner(stdout)
        for scanner.Scan() {
            m := scanner.Text()
            fmt.Println(m)
        }
    }(cmd, c)

    c <- struct{}{}
    cmd.Start()

    wg.Wait()
    fmt.Println("done.")
}

---

And Streamer sample code (just for testing):

package main

import "fmt"
import "time"

func main() {
    for i := 0; i < 10; i++ {
        time.Sleep(1 * time.Second)
        fmt.Println(i, ":", time.Now().UTC())
    }
}

---

And see func (c *Cmd) StdoutPipe() (io.ReadCloser, error) Docs:

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves; however, an implication is that it is incorrect to call Wait before all reads from the pipe have completed. For the same reason, it is incorrect to call Run when using StdoutPipe. See the example for idiomatic usage.