This error is quite strange, whenever I use Kmeans of sklearn with n_jobs > 1 in a unittest and while requiring cv2 using setuptools results in a None being called by joblib.
A minimal failing example:
setup.py:
from setuptools import setup
setup(
name = "libbla",
# Removing "cv2" resolves the issue :S
install_requires = ["numpy", "scikit-learn", "cv2"],
test_suite = 'tests'
)
tests/some_test.py:
from sklearn.cluster import KMeans
# also fails without importing sklearn and cv2, just want them for the version numbers.
import unittest, numpy, sklearn, cv2
print("cv2", cv2.__version__)
print("np", numpy.__version__)
print("skl", sklearn.__version__)
class TestFeatureCreator(unittest.TestCase):
def test_kmeans_2_features(self):
KMeans(n_clusters = 2, n_jobs = 4).fit_predict(numpy.random.randn(360000, 3))
tests/__init__.py is empty.
Then whenever I run python2.7 setup.py test, I get the following output:
$ python2.7 setup.py test
running test
Searching for cv2
Best match: cv2 1.0
Processing cv2-1.0-py2.7.egg
Using /home/herbert/Spyder/bla/.eggs/cv2-1.0-py2.7.egg
running egg_info
writing requirements to libbla.egg-info/requires.txt
writing libbla.egg-info/PKG-INFO
writing top-level names to libbla.egg-info/top_level.txt
writing dependency_links to libbla.egg-info/dependency_links.txt
reading manifest file 'libbla.egg-info/SOURCES.txt'
writing manifest file 'libbla.egg-info/SOURCES.txt'
running build_ext
('cv2', '3.0.0-dev')
('np', '1.10.1')
('skl', '0.16.1')
test_kmeans_2_features (tests.some_test.TestFeatureCreator) ... ok
----------------------------------------------------------------------
Ran 1 test in 1.156s
OK
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/dist-packages/joblib/pool.py", line 535, in <lambda>
atexit.register(lambda: delete_folder(pool_folder))
TypeError: 'NoneType' object is not callable
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/dist-packages/joblib/pool.py", line 535, in <lambda>
atexit.register(lambda: delete_folder(pool_folder))
TypeError: 'NoneType' object is not callable
$
I'm not sure whether this is a opencv2, sklearn or numpy bug, which is why I went here. Would anyone know what happens here?
Some peculiarities:
"cv2" from install_requires removes both errorsKmeans atexit.register, delete_folder and pool_folder just before the error, none of them is None.Please also comment when you can't reproduce the error :)
A fix for this issue has been merged in joblib master (to be included in the next release 0.10):