Solutions to Pythons Multiprocessing Queue buffer dead lock? How to "get" from multiprocessing Queue when its full and continue multiprocessing?

Question

This question is about multiprocessing with Python and Pythons multiprocessing Queue buffer limitations rendered by my computers OS pipe. Basically, I hit the limitation of Pythons multiprocessing Queues buffer.

Here is the my simple implementation of what i have so far

import os
from multiprocessing import Queue,Lock,Manager
def threaded_results(q,*args):
    """do something"""
    q.put(*args)

def main():
    manager = Manager()
    return_dict = manager.dict()

    cpu = os.cpu_count()
    q = Queue()
    processes = []
    for i in range(cpu):
        p = Process(target=threaded_results,args=(q,*args))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    results = [q.get() for proc in processes]

I read that i have to empty the queue first before adding back to the queue orchestrated by some thing called a semaphore. I'm considering using my own defined data structure or refactor my design of my code. The question is, are there any conventional solutions to bypass the OS level Queue buffer limitations for storing things in cache memory using Python? How to "get" multiprocessing Queue when its full and continue multiprocessing?

Answer 1

After working with the multiprocessing library for a while, I've found that the simplest way to implement a robust multiprocessing queue is to use multiprocessing.Manager objects. From the docs:

Create a shared queue.Queue object and return a proxy for it.

Rather than allocating a separate thread for flushing data through a pipe, a Manager object creates and manages a standard multithreading queue, which doesn't have to have data flushed through a Pipe (haven't looked through the source code, so I can't say for sure). This means your code can keep chugging away practically indefinitely.

None of this is free, and I've found that the managed queue operates much (almost 20x) slower than a multiprocessing queue in a simple test, though the difference isn't nearly as noticeable when the queue is integrated into a full system, due to other bottlenecks.

Using managed queues can make your IPC far more robust, and it's likely a good idea to take the performance trade-off unless you can find a way to live with the unreliability of a normal multiprocessing queue.