Running a complex command line in python

Question

I would like to call a complex command line in Python and capture its output, and I don't understand how I should be doing it:

Command line that I'm trying to run is:

cat codegen_query_output.json | jq -r '.[0].code' | echoprint-inverted-query index.bin

As far as I got is:

process = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE)
out, err = process.communicate()
print out

but this is a simple ls -a ([cmd, args]) any idea how should I run/structure my complex command line call?

Answer 1

The cleanest way is to create 2 subprocesses piped together. You don't need a subprocess for the cat command, just pass an opened file handle:

import subprocess

with open("codegen_query_output.json") as input_stream:
    jqp = subprocess.Popen(["jq","-r",'.[0].code'],stdin=input_stream,stdout=subprocess.PIPE)
    ep = subprocess.Popen(["echoprint-inverted-query","index.bin"],stdin=jqp.stdout,stdout=subprocess.PIPE)
    output = ep.stdout.read()
    return_code = ep.wait() or jqp.wait()

The jqp process takes the file contents as input. Its output is passed to ep input.

In the end we read output from ep to get the final result. The return_code is a combination of both return codes. If something goes wrong, it's different from 0 (more detailed return code info would be to test separately of course)

Standard error isn't considered here. It will be displayed to the console, unless stderr=subprocess.STDOUT is set (to merge with piped output)

This method doesn't require a shell or shell=True, it's then more portable and secure.