Search code examples
pythonsubprocessio-redirectionfilehandle

How does subprocess handle file pointer?

Suppose the code skeleton in Python looks like this. This code is invoking another executable bash script to run in parallel, on Ubuntu 18.04. 

#!/usr/bin/env python3
#encoding:utf-8

import subprocess 
with open(file='subprocess_output.log', mode='a') as file_pointer:
    subprocess.Popen(args=['./subprocess.bash'], stdout=file_pointer, stderr=file_pointer)

#Next section

It is working. But I am curious, whether the with statement is closing the file pointer and releasing the resource as soon as the main code moves on to the next section. Is not that exactly what a with statement is supposed to do? To acquire and release the resources safely? But if the handle to the log file is released immediately following the invocation, how is the subprocess buffer writing to the log later, when the main script has moved on?

Probably the answer is obvious to anyone with a deeper understanding how files are handled by the OS at a lower level. So any pointer to an online resource (pun intended) will be great.

Solution

  • Each process has its own set of open files. Subprocesses are created with copies of (a subset of) their parent’s open file handles (“descriptors” in Unix, with the subset governed by the FD_CLOEXEC flag). The with closes the parent’s handles immediately, but this doesn’t affect the child at all.

    Sometimes it matters whether any copies remain: reading from a pipe, for example, produces EOF when every copy of the writing end has been closed.