__del__ not implicitly called on object when its last reference gets deleted

Question

I have a class that starts a thread in its __init__ member and I would like join that thread when the instance of that class is not needed anymore, so I implemented that clean-up code in __del__.

It turns out that the __del__ member is never called when the last reference of the instance gets deleted, however if I implicitly call del, it gets called.

Below is a shorter modified version of my implementation that shows the issue.

import sys
from queue import Queue
from threading import Thread

class Manager:

    def __init__(self):
        ''' 
        Constructor. 
        '''
        # Queue storing the incoming messages.
        self._message_q = Queue()
        # Thread de-queuing the messages.
        self._message_thread = \
            Thread(target=process_messages, args=(self._message_q,))
        # Start the processing messages thread to consume the message queue.
        self._message_thread.start()

    def __del__(self):
        ''' 
        Destructor. Terminates and joins the instance's thread.
        '''
        print("clean-up.")
        # Terminate the consumer thread.
        # - Signal the thread to stop.
        self._message_q.put(None)
        # - Join the thread.
        self._message_thread.join()

def process_messages( message_q):
    ''' 
    Consumes the message queue and passes each message to each registered
    observer.
    '''
    while True:
        print("got in the infinite loop")
        msg = message_q.get()
        print("got a msg")
        if msg is None:
            # Terminate the thread.
            print("exit the loop.")
            break
        # Do something with message here.

mgr = Manager()
print("mgr ref count:" + str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 
#del mgr

The console outputs the following for this code:

mgr ref count:1
got in th infinite loop

The execution hangs since the thread is still running. For some reason I don't understand __del__ is not called, and as a result of that the thread is not terminated.

If I uncomment the last line del mgr to explicitly delete the instance, then __del__ gets called and the thread clean-up occurs.

mgr ref count:1
clean-up.
got in the infinite loop
got a msg
exit the loop.
Press any key to continue . . .

Does anyone have an explanation for this?

Answer 1

Silvio's answer is correct, but incomplete. In fact, it's actually guaranteed that mgr won't be deleted in this case because:

1. The message thread won't exit until mgr's deleter has been invoked, and
2. The main thread doesn't begin the part of process shutdown that clears module globals until after all non-daemon threads have completed

This ends up with a cyclic problem:

1. The thread won't complete until mgr is destroyed
2. mgr won't be destroyed until the thread completes

The explicit del mgr here works, assuming no other references to mgr exist (implicitly or explicitly). You could get a safer, and automatic version of this cleanup by putting the code in a function, e.g. a standard design (that actually makes things run quicker by replacing use of dict-based globals with function array-based locals) is to put the main functionality in a function, then invoke it:

def main():
    mgr = Manager()
    print("mgr ref count:" + str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 

if __name__ == '__main__':
    main()

It's still not perfect though; an exception can end up holding the frame object after main exits leading to non-deterministic cleanup (similarly, only the CPython reference interpreter uses reference-counting as its primary GC mechanism; on other Python interpreters cleanup is not deterministic). The only way to make this completely deterministic is to make your object a context manager, and use a with statement with it, e.g.:

import sys
from queue import Queue
from threading import Thread

class Manager:
    def __init__(self):
        # unchanged, except for one added line:
        self._closed = False  # Flag to prevent double-close

    # Give named version for rare cases where cleanup must be triggered
    # manually in some other function and therefore with statements can't be used
    def close(self):
        '''Signal thread to end and wait for it to complete'''
        if not self._closed:
            self._closed = True
            print("clean-up.")
            # Terminate the consumer thread.
            # - Signal the thread to stop.
            self._message_q.put(None)
            # - Join the thread.
            self._message_thread.join()

    __del__ = close  # Call it for users on best-effort basis if they forget to use with

    def __enter__(self):
        return self    # Nothing special to do on beginning with block
    def __exit__(self, exc, obj, tb):
        self.close()  # Perform cleanup on exiting with block

def process_messages( message_q):
    # unchanged

with Manager() as mgr:
    print("mgr ref count:" + str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 
# Guaranteed cleanup of mgr here