Signal the end of jobs on the Queue?

Question

Here's an example code of from Python documentation:

def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

I modified it to fit my use case like this:

import threading
from Queue import Queue

max_threads = 10

q = Queue(maxsize=max_threads + 2)

def worker():
  while True:
    task = q.get(1)
    # do something with the task
    q.task_done()

for i in range(max_threads):
  t = threading.Thread(target=worker)
  t.start()

for task in ['a', 'b', 'c']:
  q.put(task)

q.join()

When I execute it, debugger says that all the jobs were executed, but q.join() seems to wait forever. How can I send a signal to the worker threads that I already sent all the tasks?

Answer 1

q.join() actually returns. You can test that by put print("done") after the q.join() line.

....
q.join()
print('done')

Then, why does it not end the program? Because, by default, threads are non-daemon thread.

You can set thread as daemon thread using <thread_object>.daemon = True

for i in range(max_threads):
    t = threading.Thread(target=worker)
    t.daemon = True # <---
    t.start()

According to threading module documentation:

daemon

A boolean value indicating whether this thread is a daemon thread (True) or not (False). This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.

The entire Python program exits when no alive non-daemon threads are left.

New in version 2.6.