how to document cython function on readthedocs

On ReadTheDocs I am not allowed to compile cython extensions, is it possible to configure sphinx in order to extract the docstrings from the cython files without actually compiling them?

thanks!

Solution

I faced the same problem and found that it is now possible to compile Cython extensions on readthedocs.

Short answer: Cython modules can be compiled within the virtualenv provided by readthedocs build process.

For a slightly longer answer and an example project see below.

What is the problem?

As I have understood it, sphinx imports all modules of the project that shall be documented and then extracts the docstrings in python. This fails for Cython modules, as they cannot be directly imported and have to be compiled first. Compiling the modules does not work out of the box on readthedocs, but they are providing a tool to realize this.

Update 2024-04-02

While the underlying problem is still the same, readthedocs has changed a bit in the meantime. The option to install the package in a virtualenv is no longer a setting, but part of the buildsystem. The installation can be specified in the .readthedocs.yaml file, here using an example from the Cython documentation itself:

python:
  install:
    - requirements: docs/requirements.txt  # requirements file for the documentation build
    - method: setuptools  # runs setup.py
      path: .

Please note, that the installation using setuptools is listed as deprecated, so switching to pip and a PEP-517 compliant pyproject.toml might be a more sustainable solution.

NOTE: The rest of this answer provides background information on why this problem did arise in the first place, as well as some edge cases. I will keep that as is, please note however that it still refers to the old readthedocs interface that is no longer available.

How to solve this.

When installing the project in a virtualenv, the Cython modules will be build (into .so files) and can then be imported. This might require some external modules though (numpy for the example below and, of course, Cython). These can be specified in a pip requirements file (requirements.txt) that has to be in your repository.

Enable the option install your project inside a virtualenv under Admin -> Advanced Settings on readthedocs
Enter the path to your requirements.txt relative to the project root (docs/requirements.txt in the example below)
(If necessary change the python interpreter version)

Now your project will be installed (using python setup.py install) each time the documentation is build. The output of the setup script can be seen under Setup Output if you click on the respective build in the Builds tab on readthedocs. This is where compile time errors might show up. Mind that compiling your project might take some time.

Example project

A Python package consisting of several Cython modules, each of which has Google-style docstrings.

my_project/
    setup.py
    my_package/
        __init__.py   # imports Cython modules
        cython_module1.pyx
        cython_module2.pyx
        ...
    docs/
        requirements.txt
        Makefile
        source/
            conf.py
            index.rst
            ... #  more documentation

requirements.txt

cython>=0.20
numpy>=1.9

Caveat(s)

While trying this on my project I ran into the problem that my Cython modules could not be imported. The error message of sphinx read like:

home/docs/checkouts/readthedocs.org/user_builds/... :4: WARNING: autodoc: failed to import module 'cython_module1';...
File "/home/docs/checkouts/readthedocs.org/user_builds/.../__init__.py", ...
from .cython_module1 import CythonClass

This happened, because I used to build my documentation locally and had added a line like

...
sys.path.insert(0, os.path.abspath('../../')) # path to my_package
...

to my conf.py as suggested here. This had solved my problem when building locally (where I compiled my project using python setup.py build_ext --inplace), but when installing in the virtualenv this points to the wrong version of my_package (namely the sources, not the installed package). There sphinx can not find any .so files to import.

To solve this, just remove the line completely.

I hope this helps.