I am trying to get rid of AttributeError using mutiprocessing in python. I found out that multiprocessing does not allow local target functions and a possible solution would be to declare the local function I want to execute as a process with the global statement. I found the solution in this site: https://www.pythonpool.com/cant-pickle-local-object/. Unfortunately the solution provided by the website does not work and i still get the same error. I understand that a working solution would be to directly declare the function i want to process in the global scope, but the way I structured the code in my prject does not make this solution affordable, are there ways to target the local function directly?
import multiprocessing as mp
def main() -> None:
global func
def func() -> None:
print("Hello World")
process = mp.Process(target=func)
process.start()
if __name__ == "__main__":
main()
it is possible on linux but not on windows using python builtins, as the child must find the function definition when importing your main script, and as the definition is inside a function, the child cannot find it when it imports your main script.
the most straight forward solution is to use loky instead of python builtins as it serializes the functions by value instead of pickle's by reference serialization.
note that the joblib is a wrapper around loky in case you don't want to use loky's low level implementation.
one downside of both is that their functions are blocking, so you'll have to run them in a separate thread if you don't want you main thread to be blocked, ie: you cannot use it in a fire-and-forget pattern as you do with mp.Process and must instead use it in a pool-of-workers-and-work pattern.
from loky import ProcessPoolExecutor
def main() -> None:
global func
def func() -> None:
print("Hello World")
with ProcessPoolExecutor(1) as pool:
res = pool.submit(func)
res.result()
if __name__ == "__main__":
main()