Different behavior of go exec for different shell commands

Question

I'm trying to use different shell commands for a console go application, and for some reason the behavior is different for the following interactive shells.

This code prints result of a mongoDB query:

cmd := exec.Command("sh", "-c", "mongo --quiet --host=localhost blog")
stdout, _ := cmd.StdoutPipe()

stdin, _ := cmd.StdinPipe()
stdoutScanner := bufio.NewScanner(stdout)

go func() {
    for stdoutScanner.Scan() {
        println(stdoutScanner.Text())
    }
}()

cmd.Start()
io.WriteString(stdin, "db.getCollection('posts').find({status:'ACTIVE'}).itcount()\n")

//can't finish command, need to reuse it for other queries
//stdin.Close()
//cmd.Wait()

time.Sleep(2 * time.Second)

But the same code for Neo4J shell does not print anything:

cmd := exec.Command("sh", "-c", "cypher-shell -u neo4j -p 121314 --format plain")
stdout, _ := cmd.StdoutPipe()

stdin, _ := cmd.StdinPipe()
stdoutScanner := bufio.NewScanner(stdout)

go func() {
    for stdoutScanner.Scan() {
        println(stdoutScanner.Text())
    }
}()

cmd.Start()
io.WriteString(stdin, "match (n) return count(n);\n")

//can't finish the command, need to reuse it for other queries
//stdin.Close()
//cmd.Wait()
time.Sleep(2 * time.Second)

What is the difference? How can I make the second one work? (without closing the command)

P.S Neo4J works fine when I print directly to os.Stdout:

cmd := exec.Command("sh", "-c", "cypher-shell -u neo4j -p 121314 --format plain")

cmd.Stdout = os.Stdout

stdin, _ := cmd.StdinPipe()

cmd.Start()
io.WriteString(stdin, "match (n) return count(n);\n")

//stdin.Close()
//cmd.Wait()
time.Sleep(2 * time.Second)

Answer 1

When the input to cypher-shell is not an (interactive) terminal, it expects to read the entire input and execute it as a single script. “Entire input” means “everything until EOF”. This is typical for REPL programs: for example, python behaves like this, too.

So your Cypher code doesn’t even begin executing until you stdin.Close(). Your cmd.Stdout = os.Stdout example appears to work because stdin is implicitly closed when your Go program exits, and only then does cypher-shell execute your code and print to stdout, which is still connected to your terminal.

You should probably structure your process differently. For example, can’t you run a new cypher-shell for each query?

However, if all else fails, you can work around this by fooling cypher-shell into thinking that its stdin is a terminal. This is called a “pty”, and you can do it in Go with github.com/kr/pty. The catch is that this also makes cypher-shell print prompts and echo your input, which you will have to detect and discard if you wish to process the output programmatically.

cmd := exec.Command("sh", "-c", "cypher-shell -u neo4j -p 121314 --format plain")
f, _ := pty.Start(cmd)
stdoutScanner := bufio.NewScanner(f)
cmd.Start()

// Give it some time to start, then read and discard the startup banner.
time.Sleep(2 * time.Second)
f.Read(make([]byte, 4096))

go func() {
    for stdoutScanner.Scan() {
        println(stdoutScanner.Text())
    }
}()

io.WriteString(f, "match (n) return count(n);\n")
time.Sleep(2 * time.Second)

io.WriteString(f, "match (n) return count(n) + 123;\n")
time.Sleep(2 * time.Second)

Aside 1: In your example, you don’t need sh -c, because you’re not using any features of the shell. You can avoid the overhead of an additional shell process by running cypher-shell directly:

cmd := exec.Command("cypher-shell", "-u", "neo4j", "-p", "121314", "--format", "plain")

Aside 2: Do not discard returned error values in production code.