Suppose I have a number of items that I put in a queue for other processes to deal with. The items are rather large in memory, therefore I limit the queue size. At some point I will have no more things to put in the queue. How can I signal the other processes that the queue is closed?
One option would be to close the child processes when the queue is empty, but this relies on the queue being emptied slower than it is being filled.
The documentation of multiprocessing.Queue talks about the following method:
close()
Indicate that no more data will be put on this queue by the current process. The background thread will quit once it has flushed all buffered data to the pipe. This is called automatically when the queue is garbage collected.
Is it safe to call close while there are still items in the queue? Are these items guaranteed to be processed? How can a another processes know that the queue is closed?
This is a common scenario: how do I tell all queue consumers that no more items will be enqueued? Multiprocessing apps using POSIX message queues, datagram sockets, or even just named pipes, for example, might all face this.
The easiest thing to do here would be to enqueue a single, special "all done" message, which each consumer receives and puts() back on the queue for the next consumer to do the same.
(close() is indeed safe but inapplicable here. Any "in flight" items will be safely enqueued, but the close() doesn't tell the consumers that no more producers remain.)