Deadlock in Python garbage collection on exception

Question

I have encountered a strange situation where a program won't exit due to the way python handles exceptions. In this situation, I have an object which owns a Thread, and this Thread is only shut down when the object's __del__ method is called. However, if the program "exits" due to an exception involving this object, the exception itself will hold a reference to the object in its stack trace, which prevents the object from being deleted. Since the object isn't deleted, the Thread is never shut down, and so the program can't fully exit and hangs forever. Here is a small repro:

import threading

class A:

  def __init__(self):
    self._event = threading.Event()
    self._thread = threading.Thread(target=self._event.wait)
    self._thread.start()

  def __del__(self):
    print('del')
    self._event.set()
    self._thread.join()

def main():
  a = A()
  # The stack frame created here holds a reference to `a`, which
  # can be verified by looking at `gc.get_referrers(a)` post-raise.
  raise RuntimeError()

main() # hangs indefinitely

A workaround is to break the reference chain by eating the exception and raising a new one:

error = False
try:
  main()
except RuntimeError as e:
  error = True

if error:
  # At this point the exception should be unreachable; however in some
  # cases I've found it necessary to do a manual garbage collection.
  import gc; gc.collect()
  # Sadly this loses the stack trace, but that's what's necessary.
  raise RuntimeError()

Funnily enough, a similar issue occurs without any exceptions at all, simply by leaving a reference to a in the main module:

A()  # This is fine, prints 'del'
a = A()  # hangs indefinitely

What is going on here? Is this a python (3.10) bug? And is there a best practice for avoiding these sorts of issues? It really took me a long time to figure out what was happening!

Answer 1

Based on Python's data model:

It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

So you shouldn't terminate the thread in the __del__ function. Instead, it's recommended to explicitly set the flag to signal termination when appropriate or you can use context manager:

import threading

class A:

  def __init__(self):
    self._event = threading.Event()
    
  def __enter__(self):
      self._thread = threading.Thread(target=self._event.wait)
      self._thread.start()
      
  def __exit__(self):
      self._event.set()
      self._thread.join()

def main():
  with A() as a:
    raise RuntimeError()

main()