Search code examples
pythonprocessqueuemultiprocess

python multi process queue complete


I make a simple python program and below is source code.

# -*- coding: utf-8 -*-
  import sys
  from multiprocessing import Process, Queue

  def worker(q):
      test = q.get()
      print str(test) + ' success!'

  if __name__ == "__main__":
         test = ['a','e','f','d',1,2]
         print test

         q = Queue()
         q.put(test)

         p0 = Process(target=worker, args=(q,))
         p0.start()
         p0.join()

         print 'python exit!!'

and here is success output.

['a', 'e', 'f', 'd', 1, 2]
['a', 'e', 'f', 'd', 1, 2] success!
python exit!!

but when i run this program as shown below it just run with two processes.

# -*- coding: utf-8 -*-
  import sys
  from multiprocessing import Process, Queue

  def worker(q):
      test = q.get()
      print str(test) + ' success!'

  if __name__ == "__main__":
         test = ['a','e','f','d',1,2]
         print test

         q = Queue()
         q.put(test)

         p0 = Process(target=worker, args=(q,))
         p1 = Process(target=worker, args=(q,))
         p0.start()
         p1.start()
         p0.join()
         p1.join()

         print 'python exit!!'

its output this.

['a', 'e', 'f', 'd', 1, 2]
['a', 'e', 'f', 'd', 1, 2] success!

program does not end! what is the problem. Is there any problem with Queue? when i run using

p0 = Process(target=worker, args=(test,))

instead of q, there are no problem.


Solution

  • https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Queue.get:

    get([block[, timeout]])

    Remove and return an item from the queue. If optional args block is True (the default) and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Queue.Empty exception if no item was available within that time. Otherwise (block is False), return an item if one is immediately available, else raise the Queue.Empty exception (timeout is ignored in that case).

    So, when you call p0.start(), it will remove all items in queue. After you call p1.start(), the method q.get() must wait until an queue item available.