Variable number of SharedMemory objects managed by SharedMemoryManager in Python

Question

I would like to have a variable number of SharedMemory objects (depending on the number of consumer processes in my producer-consumer app architecture) managed by SharedMemoryManager so I am creating SharedMemory objects in the manager's "scope" in a for loop. Unfortunately, only the shared memory that is created last can be accessed by its name, the previous ones are probably somehow destroyed.

This code produces the described behavior. I am using Python 3.10.

from multiprocessing.managers import SharedMemoryManager
from multiprocessing.shared_memory import SharedMemory

consumer_process_count = 3

with SharedMemoryManager() as memory_manager:
    memory_names = []
    for i in range(consumer_process_count):
        shared_memory = memory_manager.SharedMemory(size=1)
        memory_names.append(shared_memory.name)
        print(f"Memory {shared_memory.name} created.")

    for memory_name in memory_names:
        try:
            SharedMemory(name=memory_name)
        except FileNotFoundError:
            print(f"Memory {memory_name} not found.")
        else:
            print(f"Memory {memory_name} found.")

Am I doing something wrong or is it just not possible to use SharedMemoryManager to manage variable number of SharedMemory objects?

I have a workaround solution that doesn't use SharedMemoryManager but I wanted to make my code cleaner and also use the advantage of having the context manager take care of resource cleanup whenever there is an unexpected error.

Answer 1

It appears to me that as you proceed through the loop executing shared_memory = memory_manager.SharedMemory(size=1), you are overlaying the previous value of shared_memory with the new one and since there are no longer any references to that prior shared memory instance, it is being "closed" rendering it inaccessible. I would suspect that you are able to find the last shared memory block that was allocated by name since a reference to it still exists.

The solution is to keep a reference to all shared memory blocks allocated. For example, append the shared memory block reference to a list, shared_memories thus preventing the block from being rendered inaccessible:

if __name__ == '__main__':
    from multiprocessing.managers import SharedMemoryManager
    from multiprocessing.shared_memory import SharedMemory

    consumer_process_count = 3

    with SharedMemoryManager() as memory_manager:
        shared_memories = []
        memory_names = []
        for i in range(consumer_process_count):
            shared_memory = memory_manager.SharedMemory(size=1)
            shared_memories.append(shared_memory)
            memory_names.append(shared_memory.name)
            print(f"Memory {shared_memory.name} created.")

        for memory_name in memory_names:
            try:
                SharedMemory(name=memory_name)
            except FileNotFoundError:
                print(f"Memory {memory_name} not found.")
            else:
                print(f"Memory {memory_name} found.")

Prints:

Memory wnsm_5e0842b0 created.
Memory wnsm_1a9758ac created.
Memory wnsm_e93c9e19 created.
Memory wnsm_5e0842b0 found.
Memory wnsm_1a9758ac found.
Memory wnsm_e93c9e19 found

Explanation

Looking at the library code it appears that when the shared memory manger creates a shared memory block it keeps only the name of the shared memory and not the actual reference to the multiprocessing.shared_memory instance that is returned to the caller. When there are no more references to the actual memory, which occurs in your original code, method __del__ is called on the instance, which issues a call to close() on the shared memory block. So strictly speaking, the shared memory block has not been "unlinked" by this action, but according to the documentation for close() :

Closes access to the shared memory from this instance. In order to ensure proper cleanup of resources, all instances should call close() once the instance is no longer needed. Note that calling close() does not cause the shared memory block itself to be destroyed.

So the block may still exist but it can no longer be accessed.