Open a python process using python's subprocess module

I'm trying to communicate with a different python interpreter in a python script. I wrote an object which is supposed to store the subprocess and read/write its stdin, stdout, stderr.

import subprocess
import fcntl
import os

class Python:

    def __init__(self):
        self.process = subprocess.Popen("python", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        fcntl.fcntl(self.process.stdout, fcntl.F_SETFL, os.O_RDONLY | os.O_NONBLOCK)
        fcntl.fcntl(self.process.stderr, fcntl.F_SETFL, os.O_RDONLY | os.O_NONBLOCK)

    def read(self):
        stdout = self.process.stdout.read()
        if stdout:
            print("stdout:", stdout)
        stderr = self.process.stderr.read()
        if stderr:
            print("stderr:", stderr)

    def exec(self, line):
        self.process.stdin.write(bytes((line + "\n").encode("ascii")))
        self.process.stdin.flush()

In the init function the subprocess is created and stdout, stderr are set to nonblocking mode. The read function justs prints the stdout, stderr to the screen for now and the exec function writes a line to the stdin of python and flushes it. I tested this using a simple echo script:

while True:
    print(input())

I was able to use my exec method and a little time later read the line I passed to the exec method using the read method.

My problem is that this doesn't work with the python interpreter. I tried to write everything to its stdin but it doesn't write anything to stdout, stderr.

Solution

Well I did some tests, and your code works as I expect. That means that you correctly create a python interpreter and pass commands to it, and the interpreter correctly executes them.

The problem is that output to a pipe may be buffered (and is here). If you were writing several kbytes, you could get the beginning part, but here, nothing is actually written to the pipe until the Python interpreter exits.

Way to confirm :

p = Python()
p.exec("print('foo')")
p.exec("print('bar')")
p.process.stdin.close() # force interpreter to exit ...
time.sleep(0.1) # wait for the interpreter to exit ...
p.read()

You should then get stdout: b'foo\r\nbar\r\n'

As suggested by J.F.Sebastian in its comment, the most straight way not to be bothered by buffering is simply to ask the interpreter not to buffer anything, either with the -u option or with the PYTHONUNBUFFERED environment variable :

class Python:

    def __init__(self):
        self.process = subprocess.Popen("python -u", stdin=subprocess.PIPE,
                                         stdout=subprocess.PIPE,
                                         stderr=subprocess.PIPE)
        ...