Getting pv output with subprocess

Question

I am writing a script to automate database imports in MySQL. I'm trying to write code that displays the output of pv as the database is imported:

    pv = subprocess.Popen(
        ["pv", "-f", restore_filepath],
        bufsize=1,
        stderr=subprocess.PIPE,
        stdout=subprocess.PIPE,
        universal_newlines=True,
        shell=True,
    )
    subprocess.Popen(
        [
            "mysql",
            "-u{}".format(db_user),
            "-p{}".format(db_pass),
            db_name,
        ],
        shell=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.DEVNULL,
        stdin=pv.stdout,
    )
    for line in pv.stderr:
        print("hello")
        sys.stdout.write("\r" + line.rstrip("\n"))
        sys.stdout.flush()

This is based on the code in this question, but it is not working for me. The hello never even gets printed, even if I comment out the other lines in the for loop - for line in pv.stderr is blocking and I don't know why. It never unblocks, so even once the process has completed, the program is still stuck - I have to kill it.

What am I doing wrong that is causing for line in pv.stderr to block?

Answer 1

When shell=True, args[0] is the command to be executed and args[1:] are arguments passed to sh. You want -p and restore_filepath to be passed to pv not sh. So use shell=False. The same goes for your other subprocess.Popen call.

Since with shell=True,pv is not receiving any arguments, so it is hanging because it was still waiting for a filename. So again, the solution is to use shell=False.

Also note when replacing a shell pipeline, you should close stdout on the first process to allow it to receive SIGPIPE if the second process exists. This allows some programs (those that handle SIGPIPE) to more gracefully exit if the pipeline is broken.

pv = subprocess.Popen(
    ["pv", "-f", restore_filepath],
    shell=False,  # optional, since this is the default
    bufsize=1,
    stderr=subprocess.PIPE,
    stdout=subprocess.PIPE,
    universal_newlines=True,
)
mysql = subprocess.Popen(
    ["mysql",
     "-u{}".format(db_user),
     "-p{}".format(db_pass),
     db_name],
    shell=False,
    stdout=subprocess.PIPE,
    stderr=subprocess.DEVNULL,
    stdin=pv.stdout,
)
pv.stdout.close() # Allow pv to receive a SIGPIPE if mysql exits. 
for line in pv.stderr:
    print("hello")
    sys.stdout.write("\r" + line.rstrip("\n"))
    sys.stdout.flush()