Can one get all attributes in a Python multiprocessing manager namespace?

Question

I have a multiprocessing namespace initialized as the following

from multiprocessing import Manager
manager = Manager()
manager_namespace = manager.Namespace()
manager_namespace.__setattr__('x', 1)
manager_namespace.__setattr__('y', 2)
manager_namespace.__setattr__('z', 3)

I can access these attributes just fine individually by using

manager_namespace.x

but I would like a way to be able to iterate through all set attributes so I can modify them in batches without having to type manager_namespace.(x/y/z) every time. I used VSCode to try and find out where these attributes are stored in the namespace object but couldn't find the location. If anyone has any insights, they would be much appreciated!

Answer 1

As you have probably figured out, the problem is that manager_namespace is actually a proxy object and inspecting its __dict__ attribute will not reveal any useful information. Therefore, there is no general solution to your problem that can work for an arbitrary Namespace proxy.

However, if you look at the documentation for multiprocessing.managers.Namespace, you will see (in part):

A namespace object has no public methods, but does have writable attributes. Its representation shows the values of its attributes.

However, when using a proxy for a namespace object, an attribute beginning with '_' will be an attribute of the proxy and not an attribute of the referent:

So, as long as:

1. Your attributes do not begin with an underscore ('_') but otherwise are legal variable names and
2. The values of the attributes are not values that contain an equal sign ('=') then

You should be able to take the string representation of the Namespace proxy and with a simple regex pull out all the attribute names:

from multiprocessing import Manager
import re

# required for Windows:
if __name__ == '__main__':
    manager = Manager()
    manager_namespace = manager.Namespace()
    manager_namespace.__setattr__('x', 1)
    manager_namespace.__setattr__('y', 2)
    manager_namespace.__setattr__('z', 3)

    attributes = re.findall(r'\b([A-Za-z][A-Za-z_0-9]*)=', str(manager_namespace))
    print(attributes)

Prints:

['x', 'y', 'z']

You could relax the condition for naming your attributes but would have to then adjust the regular expression accordingly.

So it is only under this extremely limited and possibly not very useful set of circumstances, assuming that you have control over them, that getting the attributes could be feasible.

Update

Another possibility is to use a new, customized managed class MyNamespace (call it whatever you want) in place of Namespace that supports a get_dict (call it whatever you want) method that will return the dictionary of the actual namespace object:

from multiprocessing.managers import BaseManager, NamespaceProxy

class MyNamespaceManager(BaseManager):
    pass

class MyNamespace:
    def get_dict(self):
        return self.__dict__

class MyNamespaceProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_dict')

    def get_dict(self):
        return self._callmethod('get_dict')

# required for Windows:
if __name__ == '__main__':
    MyNamespaceManager.register('MyNamespace', MyNamespace, MyNamespaceProxy)
    with MyNamespaceManager() as manager:
        ns = manager.MyNamespace()
        ns.__setattr__('x', 1)
        ns.__setattr__('y', 2)
        ns.z = 3 # Different style of assignment

        print(ns.get_dict())

Prints:

{'x': 1, 'y': 2, 'z': 3}

If you will be using the SyncManager class because you need other built-in managed objects, for example, dict, then:

from multiprocessing import Manager
from multiprocessing.managers import SyncManager, NamespaceProxy

class MyNamespace:

    def get_dict(self):
        return self.__dict__

class MyNamespaceProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_dict')

    def get_dict(self):
        return self._callmethod('get_dict')

# required for Windows:
if __name__ == '__main__':
    SyncManager.register('MyNamespace', MyNamespace, MyNamespaceProxy)
    with Manager() as manager:
        ns = manager.MyNamespace()
        ns.__setattr__('x', 1)
        ns.__setattr__('y', 2)
        ns.z = 3 # Different style of assignment

        print(ns.get_dict())