Search code examples
pythonmultithreadingproxymultiprocessingpython-multithreading

How to use python multiprocessing proxy objects in multithreading

I am using pyhton's multiprocessing package in a standard client-server model.
I have a few types of objects in the server that I register through the BaseManager.register method, and use from the client through proxies (based on the AutoProxy class).

I've had random errors pop up when I was using those proxies from multiple client threads, and following some reading I discovered that the Proxy instances themselves are not thread safe. See from the python multiprocessing documentation:

Thread safety of proxies
Do not use a proxy object from more than one thread unless you protect it with a lock. (There is never a problem with different processes using the same proxy.)

My scenario fits this perfectly then. OK, I know why it fails. But I want it to work :) so I seek an advice - what is the best method to make this thread-safe?
My particular case being that (a) I work with a single client thread 90% of the time, (b) the actual objects behind the proxy are thread safe, and (c) that I would like to call multiple methods of the same proxied-object concurrently.

As always, Internet, those who help me shall live on and never die! Those who do their best might get a consolation prize too.

Thanks,
Yonatan

Solution

  • Unfortunately, this issue probably doesn't relate to too many people :(

    Here's what we're doing, for future readers:

    • We'll be using regular thread locks to guard usage of the 'main' proxy
    • The main proxy provides proxies to other instances in the server process, and these are thread-specific in our contexts, so we'll be using those without a lock

    Not a very interesting solution, yeah, I know. We also considered making a tailored version of the AutoProxy (the package supports that) with built-in locking. We might do that if this scenario repeats in our system. Auto-locking should be done carefully, since this is done 'behind the scenes' and can lead to race-condition deadlocks.

    If anyone has similar issues in the future - please comment or contact me directly.