Subdirectory structure breaks C++ extension build

Question

I can't seem to get around a problem where importing a C++ extension module no longer works when a subdirectory structure is used.

The two cases below present a simple working case and a slightly altered case that I cannot for the life of me get to work.

Scenario 1 (working)

Project tree:

demo_cext/         # Current working directory for all of this
├── _cmodule.cc
└── setup.py

Contents of setup.py:

import setuptools

module = setuptools.Extension("_cmod", sources=["_cmodule.cc"], language="c++")

if __name__ == "__main__":
    setuptools.setup(name="cmod", ext_modules=[module])

Contents of _cmodule.cc, basically the hello world of C extensions, which creates a function foo() that takes no args and returns 5.

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *
foo(PyObject *self, PyObject *args) {
    /* noargs() */
    if (!PyArg_ParseTuple(args, "")) {
        return NULL;
    }
    return PyLong_FromLong(5);
}

static PyMethodDef FooMethods[] = {
    {"foo",  foo,      METH_VARARGS,  "Do the foo"},
    {NULL,   NULL,     0,             NULL}
};

PyDoc_STRVAR(module_doc, "This is the module docstring.");
static struct PyModuleDef cmodule = {
    PyModuleDef_HEAD_INIT,
    "cmod",
    module_doc,
    -1,
    FooMethods,
    NULL,
    NULL,
    NULL,
    NULL
};

PyMODINIT_FUNC
PyInit__cmod(void) {
    PyObject* m = PyModule_Create(&cmodule);
    if (m == NULL) {
        return NULL;
    }
    return m;
}

The whole thing works like a charm:

$ python3 -V
Python 3.7.4
$ python3 setup.py build install

>>> import _cmod
>>> _cmod.foo()
5

Scenario 2 (broken)

Reorienting the project a bit to a layout covered specifically in the Python docs.

$ rm -rf build/ dist/ cmod.egg-info/ && \
> mkdir cmod/ && touch cmod/__init__.py && \
> mv _cmodule.cc cmod/

Leaves me with:

demo_cext/         # Current working directory for all of this
├── cmod
│   ├── __init__.py
│   └── _cmodule.cc
└── setup.py

I change setup.py slightly:

import setuptools

module = setuptools.Extension("cmod._cmod", sources=["cmod/_cmodule.cc"], language="c++")

if __name__ == "__main__":
    setuptools.setup(name="cmod", ext_modules=[module])

Now after again running:

$ python3 setup.py build install

Trying to import the module leaves me with:

>>> from cmod import _cmod
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name '_cmod' from 'cmod' (.../demo_cext/cmod/__init__.py)
>>> import cmod._cmod
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cmod._cmod'

What do I have wrong here? I'm sure it is something simple with naming conventions. This seems to fly directly in the face of how this is all laid out in the Python docs.

Answer 1

Try changing directories before launching Python (e.g. cd ..). Your traceback indicates that you're finding cmod in demo_cext/cmod (your source directory, not the installation directory), but the built extension wouldn't be there (it would be somewhere in demo_cext/build after build, and your system site-packages directory after install).

An alternative solution, if this will only ever be used on Python 3, would be to get rid of the cmod/__init__.py file; an empty __init__.py file isn't necessary to make a package in Python 3 thanks to PEP 420. By removing the __init__.py, implicit namespace packaging should take over, and it should automatically search all cmod packages in sys.path for the submodule you're trying to import.