Search code examples
pythoninstallationcythondistutils

python setup.py install, out of memory in virtualenv


I am coding a Python package that includes some Cython code. I use virtualenv. It compiles fine when I write

$ source activate
$ python setup.py build_ext --inplace

but then, when I try to install the package, it starts to consume memory until my computer hangs (or I kill the process before). More specifically, it hangs when I try:

$ python setup.py install
running install
running bdist_egg
running egg_info
creating ColoredHRG.egg-info
writing ColoredHRG.egg-info/PKG-INFO
writing top-level names to ColoredHRG.egg-info/top_level.txt
writing dependency_links to ColoredHRG.egg-info/dependency_links.txt
writing manifest file 'ColoredHRG.egg-info/SOURCES.txt

I think there following [https://github.com/docker/docker/issues/10025][1] may be the source of the problem. Essentially, in the issue cited above, it says the following

I can replicate this issue. The above resolution does not suggest that changing Python versions fixed anything. It's the writing of files by setuptools that's causing it, though I don't know why. Sometimes it hangs for me when writing dependency_links.txt and sometimes SOURCES.txt. Will see if I can investigate further.

...

Haha, ok, the issue is that you're calling setup.py from / and distutils does a listdir('.') from the calling directory, so I assume it's walking the entire filesystem. So I guess, don't do that. https://github.com/python/cpython/blob/master/Lib/distutils/filelist.py#L245

I have an idea of what is going on, but I don't know how to solve the problem.

Further information that may be useful. Below the content of setup.py

from setuptools import setup
from Cython.Build import cythonize

setup( name = 'ColoredHRG' ,
       version = '0.1' ,
       description = 'my package.' ,
       url = 'BLA BLA' ,
       author = 'BLA BLA' ,
       author_email = 'BLA BLA' ,
       license = 'GPL3' ,
       packages = [ 'ColoredHRG' ] ,
       ext_modules = cythonize( [ "ColoredHRG/ColoredHRG.pyx" ,
                                  "ColoredHRG/Pool.pyx" , 
                                  "ColoredHRG/MC.pyx" , 
                                  "ColoredHRG/EXAMPLE_traveling_salesman.pyx" , 
                                  "ColoredHRG/MC_ColoredHRG.pyx" ] ,
                                  language = 'c++' ) ,
                                  zip_safe = False )

EDIT: fixed wrong sentences.

EDIT: added the flag cython


Solution

  • I have finally found the problem. The folder structure of the package is the following

    ColoredHRG/setup.py
    ...
    ColoredHRG/ColoredHRG/ColoredHRG.pyx
    ColoredHRG/ColoredHRG/MC.pyx
    ...
    ColoredHRG/ColoredHRG/examples/examples.py
    ...
    

    Aditionally, inside the folder examples, there is (was) a soft link (I am in Linux) to a folder that in the location

    ../../DATA
    

    i.e., the soft link was

    ColoredHRG/ColoredHRG/examples/DATA -> ../../DATA
    

    It turns out that, somehow, this causes distutils to enter in an infinite recursion loop, eats the whole memory of my computer and then hangs.

    I removed the soft-link and everything works now.

    EDIT: corrected typo.