I am interested in testing out many of the internal classes and functions defined within sklearn (eg. maybe add print statement to the treebuilder so I can see how the tree got built). However as many of the internals were written in Cython, I want to learn what is the best practices and workflows of testing out the functions in Jupyter notebook.
For example, I managed to import the Stack class from the tree._utils module. I was even able to construct it but unable to call any of the methods. Any thoughts on what I should do in order to call and test the cdef classes and its methods in Python?
%%cython
from sklearn.tree import _utils
s = _utils.Stack(10)
print(s.top())
# AttributeError: 'sklearn.tree._utils.Stack' object has no attribute 'top'
There are some problems which must be solved in order to be able to use c-interfaces of the internal classes.
First problem (skip if your sklearn version is >=0.21.x):
Until version 0.21.x sklearn used implicit relative imports (as in Python2), compiling it with Cython's language_level=3 (default in IPython3) would not work - so setting language_level=2 is needed for versions < 0.21.x (i.e. %cython -2) or even better, scikit-learn should be updated.
Second problem:
We need to include path to numpy-headers. Let's take a look at a simpler version:
%%cython
from sklearn.tree._tree cimport Node
print("loaded")
which fails with nothing saying error "command 'gcc' failed with exit status 1" - but the real reason can be seen in the terminal, where gcc outputs its error message (and not to notebook):
fatal error: numpy/arrayobject.h: No such file or directory compilation terminated.
_tree.pxd uses numpy-API and thus we need to provide the location of numpy-headers.
That means we need to add include_dirs=[numpy.get_include()] to Extension definition. There are two ways to do it in %%cython-magic, via -I option:
%%cython -I <path from numpy.get_include()>
...
or somewhat dirtier trick, exploiting that %%cython magic will add the include automatically when it sees string "numpy", by adding a comment like
%%cython
# requires numpy headers
...
is enough.
Last but not least:
Note: since 0.22 this is no longer an issue as pxd-files are included into the installation (see this).
The pxd-files must be present in the installation for us to be able to cimport them. This is the case for pxd-files from the sklearn.tree subpackage, as one can see in the local setup.py-file (given this PR, this seems to be more or less a random decision without a strategy behind):
...
config.add_data_files("_criterion.pxd")
config.add_data_files("_splitter.pxd")
config.add_data_files("_tree.pxd")
config.add_data_files("_utils.pxd")
...
but not for some other cython-extensions, in particular not for sklearn.neighbors-subpackage. Now, that is a problem for your example:
%%cython
# requires numpy headers
from sklearn.tree._utils cimport Stack
s = Stack(10)
print(s.top())
fails to be cythonized, because _utils.pxd cimports data structures from neighbors/*.pxd's:
...
from sklearn.neighbors.quad_tree cimport Cell
...
which are not present in the installation.
The situation is described with more details in this SO-post, your options to build are (as described in the link)
pip install -esetup.py-files.Another option is to ask the developers of sklearn to include pxd-files into the installation, so not only building but also distribution becomes possible.