Search code examples
pythonwindowsmultiprocessingsetuptools

How to get `setup.py test` working with `multiprocessing` on Windows?


Issue

I am trying to get setup.py test working with a package that uses multiprocessing on Windows. I have the following scenario:

  • A regular python package folder structure
  • with a package
  • a test folder and
  • a setup.py.

Running

python -m unittest

from the top directory executes my tests without complaints. However, when calling

python setup.py test

I get the well known freeze_support issue (see several questions here on SO):

RuntimeError: 
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:
        if __name__ == '__main__':
            freeze_support()
            ...
    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

However, following where to put freeze_support() in a Python script? I actually have the described guards in place in all my files, possibly in way too many.

Question 1: What do I have to change in my project for setup.py test to work with my multiprocessing using function?

Question 2: Which of my placed freeze_support() are actually needed? As far as I know, this is normally only needed when compiling a frozen executable.

MWE

Folder structure

project
│   setup.py
│
└─── my_package
│    │   __init__.py
│    │   my_package.py
│   
└─── test
│    │   __init__.py
│    │   test_it.py

The __init__.py files are empty, the other ones are:

my_package.py

import multiprocessing

def say_hi(arg):
    print('hi')

def main():
    pool = multiprocessing.Pool(1)
    pool.map(say_hi, range(1))

if __name__ == '__main__':
    multiprocessing.freeze_support()
    main()

test_it.py

import multiprocessing
import unittest

from my_package import my_package

class TestIt(unittest.TestCase):
    def test_say_hi(self):
        my_package.main()

if __name__ == '__main__':
    multiprocessing.freeze_support()

setup.py

#!/usr/bin/env python

import setuptools
import multiprocessing

setuptools.setup(
    name = "my_package"
)

if __name__ == '__main__':
    multiprocessing.freeze_support()

Solution

  • I encountered the similar problem. I found that the new process created by multiprocessing calls setuptools.setup method when the test is executed via setup.py and this affects something. (I'm not clear about the root cause...)

    In my case, this setup.py resolved that.

    #!/usr/bin/env python
    
    import setuptools
    
    if __name__ == '__main__':
        setuptools.setup(
            name = "my_package",
            ....,
        )