Getting information back from a process with a multiprocessing Queue

Question

I am trying to play around with multiprocessing and I would like to communicate between Python's main thread and a subprocess with a Queue. Here is a quick test code I wrote that should get periodically some results generated by the subprocess:

from multiprocessing import Process, Queue
import time

def calculate(queue):  
    n = 0
    while n < 10:   
        n += 1
        queue.put(n)
        time.sleep(1)
    queue.put(0)

def queue_getter(queue):
    executing = True
    while executing:     
        while queue.qsize():
            n = queue.get()
            print(n)
            if n == 0:
                executing = False
        time.sleep(0.1)
    print('done')

queue = Queue()
p = Process(target=calculate, args=(queue,))
p.start()
queue_getter(queue)
p.join()
print('DONE')

This program just hangs forever, while replacing Process with threading.Thread gives the expected result:

1
2
3
4
5
6
7
8
9
10
0
done
DONE

How to make Process behave the same way as Thread in this situation?

Answer 1

Your program works fine on POSIX (UNIX-like) systems.

However, for it to work properly on ms-windows and macOS, you will need to put the program itself inside a main block, so the file can be imported without side effects.

This is due to the way multiprocessing has to work on ms-windows and macOS. Read the programming guidelines for multiprocessing.

Modify your code like this:

from multiprocessing import Process, Queue
import time


def calculate(queue):
    n = 0
    while n < 10:
        n += 1
        queue.put(n)
        time.sleep(1)
    queue.put(0)


def queue_getter(queue):
    executing = True
    while executing:
        while queue.qsize():
            n = queue.get()
            print(n)
            if n == 0:
                executing = False
        time.sleep(0.1)
    print("done")


if __name__ == "__main__":
    queue = Queue()
    p = Process(target=calculate, args=(queue,))
    p.start()
    queue_getter(queue)
    p.join()
    print("DONE")