Python multiprocessing, PyAudio, and wxPython

I have a wxPython GUI, and would like to use multiprocessing to create a separate process which uses PyAudio. That is, I want to use PyAudio, wxPython, and the multiprocessing module, but although I can use any two of these, I can't use all three together. Specifically, if from one file I import wx, and create a multiprocessing.Process which opens PyAudio, PyAudio won't open. Here's an example:

file: A.py

import wx
import time
use_multiprocessing = True
if use_multiprocessing:
    from multiprocessing import Process as X
else:
    from threading import Thread as X
import B

if __name__=="__main__":
    p = X(target=B.worker)
    p.start()
    time.sleep(5.)
    p.join()

file: B.py

import pyaudio

def worker():
    print "11"
    feed = pyaudio.PyAudio()
    print "22"
    feed.terminate()

In all my tests I see 11 print, but the problem is that I don't see 22 for the program as shown.

If I only comment out import wx I see 22 and pyaudio loads
If I only set use_multiprocessing=False so I use threading instead, I see 22 and pyaudio loads.
If I do something else in worker, it will run (only pyaudio doesn't run)

I've tried this with Python 2.6 and 2.7; PyAudio 0.2.4, 0.2.7, and 0.2.8; and wx 3.0.0.0 and 2.8.12.1; and I'm using OSX 10.9.4

Solution

There are two reasons this can happen, but they look pretty much the same.

Either way, the root problem is that multiprocessing is just forking a child. This could be either causing CoreFoundation to get confused about its runloop*, or causing some internal objects inside wx to get confused about its threads.**

But you don't care why your child process is deadlocking; you want to know how to fix it.

The simple solution is to, instead of trying to fork and then clean up all the stuff that shouldn't be copied, spawn a brand-new Python process and then copy over all the stuff that should.

As of Python 3.4, there are actually two variations on this. See Contexts and start methods for details, and issue #8713 for the background.

But you're on 2.6, so that doesn't help you. So, what can you do?

The easiest answer is to switch from multiprocessing to the third-party library billiard. billiard is a fork of Python 2.7's multiprocessing, which adds many of the features and bug fixes from both Python 3.x and Celery.

I believe new versions have the exact same fix as Python 3.4, but I'm not positive (sorry, I don't have it installed, and can't find the docs online…).

But I'm sure that it has a similar but different solution, inherited from Celery: call billiards.forking_enable(False) before calling anything else on the library. (Or, from outside the program, set the environment variable MULTIPROCESSING_FORKING_DISABLE=1.)

_{* Usually, CF can detect the problem and call __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YO‌U_MUST_EXEC__, which logs an error message and fails. But sometimes it can't, and will wait end up waiting forever for an event that nobody can send. Google that string for more information.}

_{** See #5527 for details on the equivalent issue with threaded Tkinter, and the underlying problem. This one affects all BSD-like *nixes, not just OS X.}