Collapse multiple submodules to one Cython extension

This setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = (
    Extension('myext', ['myext/__init__.py',
                        'myext/algorithms/__init__.py',
                        'myext/algorithms/dumb.py',
                        'myext/algorithms/combine.py'])
)
setup(
    name='myext',
    ext_modules=cythonize(extensions)
)

Doesn't have the intended effect. I want it to produce a single myext.so, which it does; but when I invoke it via

python -m myext.so

I get:

ValueError: Attempted relative import in non-package

due to the fact that myext attempts to refer to .algorithms.

Any idea how to get this working?

Solution

First off, I should note that it's impossible to compile a single .so file with sub packages using Cython. So if you want sub packages, you're going to have to generate multiple .so files, as each .so can only represent a single module.

Second, it doesn't appear that you can compile multiple Cython/Python files (I'm using the Cython language specifically) and link them into a single module at all.

I've tried to compile multiply Cython files into a single .so every which way, both with distutils and with manual compilation, and it always fails to import at runtime.

It seems that it's fine to link a compiled Cython file with other libraries, or even other C files, but something goes wrong when linking together two compiled Cython files, and the result isn't a proper Python extension.

The only solution I can see is to compile everything as a single Cython file. In my case, I've edited my setup.py to generate a single .pyx file which in turn includes every .pyx file in my source directory:

includesContents = ""
for f in os.listdir("src-dir"):
    if f.endswith(".pyx"):
        includesContents += "include \"" + f + "\"\n"

includesFile = open("src/extension-name.pyx", "w")
includesFile.write(includesContents)
includesFile.close()

Then I just compile extension-name.pyx. Of course this breaks incremental and parallel compilation, and you could end up with extra naming conflicts since everything gets pasted into the same file. On the bright side, you don't have to write any .pyd files.

I certainly wouldn't call this a preferable build method, but if everything absolutely has to be in one extension module, this is the only way I can see to do it.