Python Multiprocessing Can't pickle local object

Question

i have read a little about multiprocessing and pickling problems, I have also read that there are some solutions but I don't really know how can they help to mine situation.

I am building Test Runner where I use Multiprocessing to call modified Test Class methods. Modified by metaclass so I can have setUp and tearDown methods before and after each run test.

Here is my Parent Metaclass:

class MetaTestCase(type):

def __new__(cls, name: str, bases: Tuple, attrs: dict):
    
    def replaced_func(fn):
        def new_test(*args, **kwargs):
            args[0].before()
            result = fn(*args, **kwargs)
            args[0].after()
            return result

        return new_test

    # If method is found and its name starts with test, replace it
    for i in attrs:
        if callable(attrs[i]) and attrs[i].__name__.startswith('test'):
            attrs[i] = replaced_func(attrs[i])

    return (super(MetaTestCase, cls).__new__(cls, name, bases, attrs))

I am using this Sub Class to inherit MetaClass:

class TestCase(metaclass=MetaTestCase):

def before(self) -> None:
    """Overridable, execute before test part."""
    pass

def after(self) -> None:
    """Overridable, execute after test part."""
    pass

And then I use this in my TestSuite Class:

class TestApi(TestCase):

def before(self):
    print('before')

def after(self):
    print('after')

def test_api_one(self):
    print('test')

Sadly when I try to execute that test with multiprocessing.Process it fails on

AttributeError: Can't pickle local object 'MetaTestCase.__new__.<locals>.replaced_func.<locals>.new_test'

Here is how I create and execute Process:

module = importlib.import_module('tests.api.test_api')  # Finding and importing module
object = getattr(module, 'TestApi')  # Getting Class from module

process = Process(target=getattr(object, 'test_api_one')) # Calling class method
process.start()
process.join()

I tried to use pathos.helpers.mp.Process, it passes pickling phase I guess but has some problems with Tuple that I don't understand:

Process Process-1:
Traceback (most recent call last):
    result = fn(*args, **kwargs)
IndexError: tuple index out of range

Is there any simple solution for that so I can pickle that object and run test sucessfully along with my modified test class?

Answer 1

As for your original question of why you are getting the pickling error, this answer summarizes the problem and offers solutions (similar to those already provided here).

Now as to why you are receiving the IndexError, this is because you are not passing an instance of the class to the function (the self argument). A quick fix would be to do this (also, please don't use object as a variable name):

module = importlib.import_module('tests.api.test_api')  # Finding and importing module
obj = getattr(module, 'TestApi')
test_api = obj()  # Instantiate!

# Pass the instance explicitly! Alternatively, you can also do target=test_api.test_api_one
process = Process(target=getattr(obj, 'test_api_one'), args=(test_api, ))  
process.start()
process.join()

Ofcourse, you can also opt to make the methods of the class as class methods, and pass the target function as obj.method_name.

Also, as a quick sidenote, the usage of a metaclass for the use case shown in the example seems like an overkill. Are you sure you can't do what you want with class decorators instead (which might also be compatible with the standard library's multiprocessing)?