Python - Multiprocessing and variables handover to each call (but in external function)

Question

I would like to enhance question from In Python, how do I pass local variable values to multiprocessing library's Pool?

where I have similar problem as in related question, just enhanced that I call it from different file.

To example: File1 Code:

import Code

result = Code.Main_Code()
print(result)

File2 Code:

from multiprocessing import Pool

def Multicore_Patterns_List_processor(args):

    Variable1, Variable2, Variable3, Pattern = args
    print(f"{Variable1=}, {Variable2=}, {Variable3=}, current pattern is {Pattern}")

def gen_args(Variable1, Variable2, Variable3):
    for p in range(10):
        yield Variable1, Variable2, Variable3, p

def Main_Code():
    Variable1 = 1
    Variable2 = 2
    Variable3 = 3
    MultiCores = 3
    print("Finish of simple code and start of heavy code in parralel")
    with Pool(processes=MultiCores) as pool:
        pool.imap(Multicore_Patterns_List_processor, gen_args(Variable1, Variable2, Variable3))
        pool.close()
        pool.join()

    print("Finish of heavy code and continue of simple code")
    done = True
    return done

This is not working.

I'm trying to reach the state where I let python by processed by single processor, because code is simple and later in the "Code.py" file there is an heavier code which allow be processed in parallel and thanks to this fact to speed time for Local function be processed (as the function is processed several times 10). After this python should continue in single processor to finish simple code.

UPD: as Johny writes I didn't put the error here, sorry. So I also tried to run your correction, but lest focus on issue. I tried to record video during debugging where I sure ".map" instead of ".imap":

link: https://youtu.be/jIJnKaUrtRU

when I use the ".imap" I don't receive any error, code just skip the code in "Multicore_Patterns_List_processor" at all as nothing prints on terminal.

Answer 1

I would propose to change the code a bit. Instead of imap use map. And you don't need to do join and close if you are running Pool inside context (inside with statement). Otherwise it works fine.

from multiprocessing import Pool, cpu_count
...

    MultiCores = cpu_count()
    print("Finish of simple code and start of heavy code in parralel")
    with Pool(processes=MultiCores) as pool:
        res = pool.map(Multicore_Patterns_List_processor, 
                       gen_args(Variable1, Variable2, Variable3))

UPD: Another thing that was kind of obvious, but I've missed to note in my answer is that the code of the main thread is executed without being guarded by __name__ == "main" statement. When Python tries to create new processes it copies all your modules to each thread. After the modules being copied they get imported and execute everything that is not guarded by if __name__ == "main": statement. So thread that was basically created a moment ago starts triggering creation of other threads and then this process gets into an infinite loop.

Below is the fix your issue:

# Code_main.py
import Code


if __name__ == "__main__":
    result = Code.Main_Code()
    print(result)