I have a function that yields from a context manager:
def producer(pathname):
with open(pathname) as f:
while True:
chunk = f.read(4)
if not chunk:
break
yield chunk
It is not a problem when the generator is entirely consumed since, during the last iteration, the generator resumes execution after the yield statement, and the loop breaks and we nicely exit the context manager.
However, if the generator is only partially consumed, and there are no more consumers to consume it entirely, will the generator remain suspended forever? In that case, we will never exit from the context manager. Would that mean the file will remain open for the rest of the program execution? Or at least until the generator is garbage collected? Is this a corner case I should take care of by myself, or can I rely on the Python runtime to close dangling context manager in time?
If you fail to consume the whole generator, the context manager won't be cleaned until the generator is garbage collected, which may take quite a while if reference cycles are involved, or you're running on a non-CPython interpreter.
You can work around this by close-ing the generator-iterator; all generator functions provide a close method on the resulting generator-iterator that raises GeneratorExit inside it; the exception bubbles out of with statements and the like to make sure they're properly cleaned deterministically.
To make it occur at a guaranteed point in time, you can use contextlib.closing to get guaranteed closing of the generator itself:
from contextlib import closing
with closing(producer(mypath)) as produced_items:
for item in produced_items:
# Do stuff, maybe break loop early
Even if you break, return, or raise an exception, the with controlling produced_items will close it, which will in turn invoke cleanup for the with statements within it.