Search code examples
pythonmultiprocessingflask-restful

multiprocessing.Process() doesn't work when targeting a method directly

Why does the following start() method of multiprocessing.Process not work when targeting the method Flask.run() and an AttributeError is raised when initialising Flask:

if __name__ == "__main__":
    Process(target=app.run,
            kwargs=dict(host="0.0.0.0",
                        port=5002,
                        ssl_context=('./cert/cert.pem', './cert/cert-key.pem'),
                        debug=False)
            ).start()
    app.run(host="0.0.0.0", port=5001)

with the following error/Traceback:

Traceback (most recent call last):
  File "projectDir/main.py", line 80, in <module>
    Process(target=app.run,
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Flask.__init__.<locals>.<lambda>'

But referencing the Flask.run() in a function, and passing that function under target= parameter does work:

def run_https(**kwargs):
    app.run(**kwargs)

if __name__ == "__main__":
    Process(target=run_https,
            kwargs=dict(host = "0.0.0.0",
                        port=5002,
                        ssl_context=('./cert/cert.pem', './cert/cert-key.pem'),
                        debug=False)
            ).start()
    app.run(host="0.0.0.0", port=5001)

More context: I am trying to encrypt the communication with a local SocketServer using flask_restful, but because certificates management might take a while on all client devices, I have decided to run two Flask() instances on two processes and two ports, one for http and another for https

Thanks in advance!

Solution

  • Since your child process's target is app.run, Python needs to send the entire app object to the new address space by serializing it with pickle and then de-serializing the result in the new process's address space. Unfortunately, the app object cannot be serialized. There can be any number of reasons but it comes down to it having one or more attributes that cannot be serialized because the nature of that attribute is such that it only is valid in the main process's address space. For example, the object's implementation might include a reference to an absolute memory location or file descriptor, which would only be valid in the address space that has created the object. However, if this same code were run in a platform that uses OS fork to create new processes, then pickle would not be used and the program would run. This is because the new child's address space inherits all the memory (read only/copy on write) and open file descriptors. See below as an example:

    def worker(f):
    print(f.read())

if __name__ == '__main__':
    from multiprocessing import Process

    with open('test.txt', 'r') as f:
        print(type(f))
        Process(target=worker, args=(f,)).start()

    Prints:

    <class '_io.TextIOWrapper'>
Traceback (most recent call last):
  File "C:\Booboo\test\test.py", line 8, in <module>
    Process(target=worker, args=(f,)).start()
  File "C:\Program Files\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python38\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python38\lib\multiprocessing\spawn.py", line 116, in spawn_main
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python38\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object