Why are my subprocesses running sequentially, rather than all at once?

Question

My script has two loops:

• in the first one multiple ssh-processes are launched, one for each remote machine;
• in the second I collect and print each process' stderr, stdout, and examine its returncode

I see, that they all run in sequence -- confirmed by printed output of the date-commands on each -- whereas I expected them to run in parallel... What am I doing wrong?

pipes = {}
myhostname = os.uname()[1].lower()
for node in Nodes:
    if node.lower() == myhostname:
        cmd = '/bin/bash'
    else:
        cmd = ('ssh -q '
            '-o PasswordAuthentication=no '
            '-o StrictHostKeyChecking=no '
            '%s /bin/bash') % node
    pipe = subprocess.Popen(cmd.split(' '),
        stdin = subprocess.PIPE, stdout = subprocess.PIPE,
        stderr = subprocess.PIPE, close_fds = True,
        encoding = locale.getlocale()[1])
    pipe.stdin.write("""
        date
        sleep 1
        date
        exit""")
    pipes[node] = pipe

logging.info('%d processes spawn, collecting outputs', len(pipes))
errCount = 0
for (node, pipe) in pipes.items():
    out, err = pipe.communicate()
    if err:
        logging.warn('%s stderr: %s', node, err.strip())
    if out:
        logging.info('%s stdout: %s', node, out.strip())
    if pipe.returncode != 0:
        errCount += 1
        logging.error('Got exit code %d from %s, increasing '
            'error-count to %d', pipe.returncode, node, errCount)

Answer 1

The commands you’re writing to each process are waiting in your process’s output buffers until you call communicate on each, flushing and closing each one’s input. You do that in sequence, so the date commands run in sequence, even though the ssh processes are all running the entire time in parallel. With this little data, you can add pipe.stdin.flush() immediately after the write to get the desired effect. (See Allan Wind’s answer for the general case for more (bytes of) commands.)