python: multiprocessing.Pipe and redirecting stderr on Windows

Question

I've a main process where I open a multiprocessing.Pipe(False) and send the writing end to a worker Process. Then, in the worker process, I run a Java program using subprocces.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE). I need to redirect the error of this subprocess to the writing end of multiprocessing.Pipe

For this I referred to this answer by Ilija as this is exactly what I want to achieve, but on my machine(Windows), it throws OSError: [Errno 9] Bad file descriptor

Machine details:

OS - Windows 10 (64bit)

Python version - 3.7.4

Code:

Method 1 (Ilija's answer)

def worker(w_conn):
    os.dup2(w_conn.fileno(), 2)
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    sp.wait()
    w_conn.close()


def main():
    r_conn, w_conn = multiprocessing.Pipe(False)
    process = multiprocessing.Process(target=worker, args=(w_conn,))
    process.start()
    
    while not r_conn.poll() and not w_conn.closed:
        # Do stuff
    else:
        # Read error from r_conn, and handle it
    
    r_conn.close()
    process.join()

if __name__=='__main__':
    main()

Error:

Process Process-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\Desktop\Workspace\Error.py", line 14, in worker
    os.dup2(w_conn.fileno(), 2)
OSError: [Errno 9] Bad file descriptor

Method 2: In worker function, sending w_conn as argument to Popen

def worker(w_conn):
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
    sp.wait()
    w_conn.close()

Error:

Process Process-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\Desktop\Workspace\Error.py", line 13, in worker
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
  File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 728, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 1077, in _get_handles
    errwrite = msvcrt.get_osfhandle(stderr.fileno())
OSError: [Errno 9] Bad file descriptor

Is there any workaround/alternate method to achive this on Windows?

Answer 1

I still don't know why "Method 1" is not working. Any information regarding this will be appreciated.

"Method 2" is wrong altogether as we can't use Connection object (returned by multiprocessing.Pipe()) as a file handle in subprocess.Popen.

What works is checking for data in stderr of subprocess sp and sending the data through w_conn to main process.

def worker(w_conn):
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    sp.wait()
    if sp.stderr.seek(0, io.SEEK_END)>0:
        w_conn.send(sp.stderr.read())
    w_conn.close()