Python multiprocessing code can only be run in terminal or debug mode

I can only run the following code by executing it in terminal or by runing it in the debug mode.

It seems that it is something about the portability issue (I guess), but can someone explain it?

I saw some tutorial emphasising the importance of if __name__ == '__main__':, but I, as a python rookie, can hardly understand.

Thanks a lot in advance.

import multiprocessing as mp


def add_abc(a, b, c):
    print(a + b + c)


def main():

    # Create a list of information
    a = [1, 2, 3, 4, 5]
    b = [6, 7, 8, 9, 10]
    c = [11, 12, 13, 14, 15]

    processes = []

    # Create a list of processes
    for i in range(5):
        p = mp.Process(target=add_abc, args=(a[i], b[i], c[i]))
        processes.append(p)

    # Start the processes
    for p in processes:
        p.start()

    # Join the processes
    for p in processes:
        p.join()


if __name__ == '__main__':

    main()

If I run it in the pycharm's normal Run mode (Shift + F10), I get EOFError and _pickle.PicklingError: (Just part of Error message)

  File "D:\Program Files\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\Program Files\Python311\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function add_abc at 0x0000023408E3C540>: attribute lookup add_abc on __main__ failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Program Files\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Program Files\Python311\Lib\multiprocessing\spawn.py", line 130, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input

Normal Output should be:

Solution

This is probably an IDE problem. Python has 2 modes of operation: normal and interactive. Think of normal mode as running a ".py" file from a terminal, and interactive as typing code into a prompt one line at a time. Multiprocessing also has two different modes of operation that are impacted by this normal and interactive operation: Spawning and Forking. Forking is not available on Windows, and is not the default on MacOS. This problem sounds like a bad interaction between running python in interactive mode while using multiprocessing in spawning mode.

Why you need if __name__ == "__main__":

When you attempt to "spawn" a new process, it will attempt to import the main file from which it was created. If importing the script is just like running it, then importing the script will cause it to try and create its own child process which will import - create a new process - import... causing an infinite loop which could create infinite child processes. This is why we use if __name__ == "__main__": to prevent the script from creating any child processes (or doing anything really other than defining functions classes and maybe some static data). __name__ is a special variable which will be "__main__" only when you run that file. If the file is imported, it will be the name of the "module" (the file name in this case: "test").

What I think the actual error message is:

The main issue is also related to importing if you are using interactive mode. Depending on the IDE you may not even realize you're using interactive mode, but if you are then python doesn't have a "main" file to import when the child process starts up. Basically when a new process is started via "spawn" it knows nothing about its parent process. It only has a "pipe" to communicate with the parent, and it knows that it needs to follow the standard python "spawn" startup procedure (from: python\lib\multiprocessing\spawn.py). The parent process will attempt to send the info needed over that pipe for the child to execute what you want, but there's one more small detail. The communication protocol used ("Pickle") cannot send functions or class definitions directly. Instead it sends instructions on how to import the correct definitions. Finally we arrive at the issue: there is no ".py" file to import if you defined the function in interactive mode. Sometimes you can get around this by putting the target function in a separate file which can be imported, and sometimes IDE's do some dark magic to get it to work in a non-standard way. Typically I simply prefer to use an IDE that only executes code from a file, and I only use the interactive mode for small experiments.