Search code examples
pythoncythoncythonize

Cython cimport cannot find .pxd module

Assume a package is structured as:

Some_Package/ 
    some_package/
        __init__.py
        core/
            __init__.py
            definition.pxd
        helper/
            __init__.py
            helper.pxd
            helper.pyx
    setup.py

Where in definition.pxd I have:

import numpy as np
cimport numpy as np
# ...
ctypedef np.int32_t INT_t

And in helper.pxd I have:

cimport some_package.core.definition
from some_package.core.definition cimport INT_t
# ...

In helper.pyx I didn't cimport anything. I configured setup.py as:

ext_modules=cythonize('./some_package/helper/helper.pyx', include_dirs=['.', './some_package/core'])

Now my problem is with python setup.py build_ext --inplace I can build successfully into .so, but when I tried to import some_package.helper.helper I got an ImportError:

ImportError: No module named "some_package.helper.helper"

I have looked into helper.cpp and found some lines like:

  __pyx_t_1 = __Pyx_ImportModule("some_package.core.definition"); if (!__pyx_t_1) __PYX_ERR(0, 1, __pyx_L1_error)

I guess this might have something to do with import path, but I cannot spot what was wrong. All __init__.py's are empty and I have imported absolute_import in every file. I also changed the include_dirs, but still not working.

Edit #1

According to the documentation, include_dirs adds to the *.pxd search path. So I also tried to change the cimport statements in helper.pxd as:

cimport definition
from definition cimport INT_t

This time, cython cannot compile: "definition.pxd" not found. However, it should be in the search path.

Edit #2

A quick workout is adding an empty definition.pyx in core/, then configure extensions as:

extensions = [
    Extension("some_package.core.definition", ["some_package/core/definition.pyx"])
    Extension("some_package.helper.helper", ["some_package/helper/helper.pyx"])
]

Then in setup.py:

ext_modules=cythonize(extensions)

Now cimport some_package.core.definition is working in helper.pxd.

However, this is not elegant.

Solution

  • I overlook that I had some lines as:

    cdef INT_t some_int = 1

    However, in the .pxd file, there cannot be any executable code. In this case, it seems Cython treats it as a package, which is not as there is .pyx file.

    There might be two methods to work around:

    1. cdef extern from a C header.

    2. Wrap-up to inline functions.