Continuous communication between parent and child subprocess in Python (Windows)?

Question

I have this script:

import subprocess

p = subprocess.Popen(["myProgram.exe"],
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE)
while True:
    out, _ = p.communicate(input().encode())
    print(out.decode())

which works fine until the second input where I get:

ValueError: Cannot send input after starting communication

Is there a way to have multiple messages sent between the parent and child process in Windows ?

[EDIT]
I don't have access to the source code of myProgram.exe
It is an interactive command line application returning results from queries
Running >> myProgram.exe < in.txt > out.txt works fine with in.txt:

query1;
query2;
query3;

Answer 1

Interacting with another running process via stdin/stdout

To simulate the use case where a Python script starts a command line interactive process and sends/receives text over stdin/stdout, we have a primary script that starts another Python process running a simple interactive loop.

This can also be applied to cases where a Python script needs to start another process and just read its output as it comes in without any interactivity beyond that.

primary script

import subprocess
import threading
import queue
import time

if __name__ == '__main__':

    def enqueue_output(outp, q):
        for line in iter(outp.readline, ''):
            q.put(line)
        outp.close()

    q = queue.Queue()

    p = subprocess.Popen(["/usr/bin/python", "/test/interact.py"],
                         stdin    = subprocess.PIPE,
                         stdout   = subprocess.PIPE,
                         # stderr   = subprocess.STDOUT,
                         bufsize  = 1,
                         encoding ='utf-8')

    th = threading.Thread(target=enqueue_output, args=(p.stdout, q))
    th.daemon = True

    th.start()

    for i in range(4):

        print("dir()", file=p.stdin)

        print(f"Iteration ({i}) Parent received: {q.get()}", end='')
        # p.stdin.write("dir()\n")
        # while q.empty():
        #     time.sleep(0)
        # print(f"Parent: {q.get_nowait()}")

interact.py script

if __name__ == '__main__':

    for i in range(2):

        cmd = raw_input()

        print("Iteration (%i) cmd=%s" % (i, cmd))

        result = eval(cmd)

        print("Iteration (%i) result=%s" % (i, str(result)))

output

Iteration (0) Parent received: Iteration (0) cmd=dir()
Iteration (1) Parent received: Iteration (0) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i']
Iteration (2) Parent received: Iteration (1) cmd=dir()
Iteration (3) Parent received: Iteration (1) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i', 'result']

This Q&A was leveraged to simulate non-blocking reads from the target process: https://stackoverflow.com/a/4896288/7915759

This method provides a way to check for output without blocking in the main thread; q.empty() will tell you if there's no data. You can play around with blocking calls too using q.get() or with a timeout q.get(2) - the parameter is number of seconds. It can be a float value less than zero.

Text based interaction between processes can be done without the thread and queue, but this implementation gives more options on how to retrieve the data coming back.

The Popen() parameters, bufsize=1 and encoding='utf-8' make it possible to use <stdout>.readline() from the primary script and sets the encoding to an ascii compatible codec understood by both processes (1 is not the size of the buffer, it's a symbolic value indicating line buffering).

With this configuration, both processes can simply use print() to send text to each other. This configuration should be compatible for a lot of interactive text based command line tools.