Why do we get a NameError when trying to use the SharedMemoryManager (python 3.8) as a replacement for the BaseManager?

Question

Python 3.8 introduces new shared memory features. We are trying to use the SharedMemoryManager and a NameError is thrown.

I thought that we might do something wrong in our complex scenario, so I broke it down using python documentation snippets.

try:
    # python >= 3.8
    from multiprocessing.managers import SharedMemoryManager as Manager
except:
    # python < 3.8
    from multiprocessing.managers import BaseManager as Manager

class MathsClass:
    def add(self, x, y):
        return x + y
    def mul(self, x, y):
        return x * y

class MyManager(Manager):
    pass

MyManager.register('Maths', MathsClass)

if __name__ == '__main__':
    with MyManager() as manager:
        maths = manager.Maths()
        print(maths.add(4, 3))         # prints 7
        print(maths.mul(7, 8))         # prints 56

This is pretty much taken from the multiprocessing docs (except for the fallback import) and works fine in python 3.7 but throws the following error in python 3.8:

Traceback (most recent call last):
  File "scripts/debug_shared_memory_issue.py", line 21, in <module>
    maths = manager.Maths()
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 740, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 625, in _create
    id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 91, in dispatch
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 210, in handle_request
    result = func(c, *args, **kwds)
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 1312, in create
    if hasattr(self.registry[typeid][-1], "_shared_memory_proxy"):
NameError: name 'self' is not defined
---------------------------------------------------------------------------

The release notes of python 3.8 and the documentation say that SharedMemoryManager is "A subclass of BaseManager" so we expected it to work as a drop-in replacement. But it doesn't seem so. What are we doing wrong? Looking at the current 3.8 branch of CPython there don't seem to be relevant changes to this. In Python 3.9 there is an explicit self arg in the create function though. But 3.9 is WIP, so we'd rather not use it in production software.

Thanks for your help!

Answer 1

It was a bug fixed in python/cpython@142566c (v3.9.0a1).

You can patch from python/cpython/blob/v3.9.0a1/Lib/multiprocessing/managers.py#L1269-L1277:

from multiprocessing.managers import SharedMemoryManager as Manager

import sys

if sys.version_info < (3, 9):
    from multiprocessing.managers import Server, SharedMemoryServer

    def create(self, c, typeid, /, *args, **kwargs):
        if hasattr(self.registry[typeid][-1], "_shared_memory_proxy"):
            kwargs['shared_memory_context'] = self.shared_memory_context
        return Server.create(self, c, typeid, *args, **kwargs)

    SharedMemoryServer.create = create