Attempting to run threads concurrently inside while loop in Python 3.11

Question

I am trying to get my head around threading in Python 3.11 and I am trying to work out why when I put a time.sleep(120) inside execute_subtasks that the next thread is not processed and the code appears to run sequentially instead of concurrently.

Do I need to start the thread outside of the for loop or do I need to move the location of the join?

from threading import Thread, Event, active_count, current_thread

threads = list()
while True:
    try:
        # ignore the main thread
        if active_count() > 1:
            for index, thread in enumerate(threads):
                thread.join()

        for message in queue.get_messages(4):
            if active_count() >= config.maxthreads + 1:
                # Thread pool is about to overflow. Skipping.
                continue

            x = threading.Thread(
                target=message.get_task().execute_subtasks,
                daemon=True,
                args=[message.get_context()]
            )
            threads.append(x)
            x.start()
    except Exception:
        # Exception in dispatch loop

Answer 1

You do not need to use active_count() when joining dead threads because the only threads you're interested in are in your threads list.

Implement this function...

def remove_dead_threads(_list):
    for thread in [t for t in _list if not t.is_alive()]:
        thread.join()
        _list.remove(thread)

Then, as the first line of code after try: do...

remove_dead_threads(threads)

This will clean up your threads list by joining those that are no longer alive.

You could also consider using a ThreadPoolExecutor to make pool management easier