Search code examples
pythonpython-multiprocessingconcurrent.futures

Getting "Queue objects should only be shared between processes through inheritance" but I'm not using a Queue


I am trying to use a ProcessPoolExecutor, but I am getting the error "Queue objects should only be shared between processes through inheritance", but I am not using a Queue (at least not explicitly). I can't find anything that explains what I am doing wrong.

Here is some code that demonstrates the issue (not my actual code):

from concurrent.futures import ProcessPoolExecutor, as_completed

class WhyDoesntThisWork:

    def __init__(self):
        self.executor = ProcessPoolExecutor(4)

    def execute_something(self, starting_letter):
        futures = [self.executor.submit(self.something, starting_letter, d) for d in range(4)]
        letter = None
        for future in as_completed(futures):
            letter = future.result()
        print(letter)

    def something(self, letter, d):
        # do something pointless for the example
        for x in range(d):
            letter = chr(ord(letter) + 1)

if __name__ == '__main__':
    WhyDoesntThisWork(). execute_something('A')

El Ruso has pointed out that making something() a staticmethod or classmethod makes the error go away. Unfortunately, my actual code needs to call other methods using self.


Solution

  • try this code for something

    @staticmethod
    def something(letter, d):
        # do something pointless for the example
        for x in range(d):
            letter = chr(ord(letter) + 1)
    

    or refactor to:

    from concurrent.futures import ProcessPoolExecutor, as_completed
    
    
    class WhyDoesntThisWork:
        def something(self, letter, d):
            # do something pointless for the example
            for x in range(d):
                letter = chr(ord(letter) + 1)
            return letter
    
    
    if __name__ == '__main__':
        executor = ProcessPoolExecutor(4)
        letter = 'A'
        obj = WhyDoesntThisWork()
        futures = [executor.submit(obj.something, letter, d) for d in range(4)]
        for future in as_completed(futures):
            print(future.result())