Poetry + Cython + tests (Nosetests)

Question

I use Poetry to build my package with cython extensions. Now I'd like to write tests for it (preferably with nosetest). The problem is that I need to precompile binaries what is usually done with setup.py build_clib build_ext --inplace

The best solution for me is to run tests without creating extra .py or .sh files in the directory as I already have build.py. It is ok to run tests after installing the package in virtual environment, like it is implemented on the readthedocs server.

I also got familiar with taskipy, so some bash commands in my pyproject.toml are also ok. Any other packages that work with pyproject.toml are welcome.

Maybe there are any hooks for Poetry, as it cythonizes and comiles while creating .whl distribution file.

Any help on this will be appreciated.

UPD Tox looks like suitable tool, but it does not see pyproject.toml while it in the directory. Links to repos with tox and cython in packages or tutorials are extremely welcome.

Answer 1

If the extension is part of the distribution, you don't need to do anything besides running poetry install - poetry will build the extensions in-place as part of the editable installation of your project.

In other cases, you can embed calling distutils commands in your tests as part of suite setup/teardown. I'm not very familiar with nose, but here's a simple example. Imagine I have a fib.pyx (this is an example from the Cython book):

def fib(long n):
    '''Returns the nth Fibonacci number.'''
    cdef long a=0, b=1, i
    for i in range(n):
        a, b = a + b, a
    return a

A test_fib.py module that builds the fib library and removes it on tests success:

from distutils.dist import Distribution
from distutils.core import Extension
from pathlib import Path
from Cython.Build import cythonize


fib_source = Path('fib.pyx')

# distutils magic. This is essentially the same as calling
# python setup.py build_ext --inplace
dist = Distribution(attrs={'ext_modules': cythonize(fib_source.name)})
build_ext_cmd = dist.get_command_obj('build_ext')
build_ext_cmd.ensure_finalized()
build_ext_cmd.inplace = 1
build_ext_cmd.run()

fib_obj = Path(build_ext_cmd.get_ext_fullpath(fib_source.stem))

# the lib was built, so the import will succeed now
from fib import fib


def teardown_module():
    # remove built library
    fib_obj.unlink()

    # if you also want to clean the build dir:
    from distutils.dir_util import remove_tree
    remove_tree(build_ext_cmd.build_lib)
    remove_tree(build_ext_cmd.build_temp)


# sample tests

def test_zero():
    assert fib(0) == 0


def test_ten():
    assert fib(10) == 55

You are probably customizing the setup_kwargs in the custom build.py. To reuse this code, adapt the dist initialization, for example:

from build import build

setup_kwargs = {}
build(setup_kwargs)
dist = Distribution(attrs=setup_kwargs)
...

pytest example

Things can be organized a lot more conveniently with pytest. Create a file named conftest.py with the setup/teardown code extracted to hooks:

# conftest.py

from distutils.core import Extension
from distutils.dist import Distribution
from distutils.dir_util import remove_tree
from pathlib import Path
from Cython.Build import cythonize


def pytest_sessionstart(session):
    fib_source = Path('fib.pyx')
    dist = Distribution(attrs={'ext_modules': cythonize(fib_source.name)})
    build_ext_cmd = dist.get_command_obj('build_ext')
    build_ext_cmd.ensure_finalized()
    build_ext_cmd.inplace = 1
    build_ext_cmd.run()
    session.fib_obj = Path(build_ext_cmd.get_ext_fullpath(fib_source.stem))


def pytest_sessionfinish(session):
    session.fib_obj.unlink()

Now the tests become a lot cleaner and the setup code is run once for the whole test session. The above tests example, revisited:

from fib import fib


def test_zero():
    assert fib(0) == 0


def test_ten():
    assert fib(10) == 55