Dynamic on the fly compile

Question

I have individual Functions (from user input) for plots (x^2, e^x, x, ...) and I would compute the coordinates in cython. I created with exec a individual function in Python code yet, but thats to slow for me.

I tried to generate individual .pyx files and compile them to .pyd files (on Windows).

This is my way to generate the individual .pyx file and compile them. The generating works fine, but I only can compile them when I hadn't imported the cython module from the .pyd file in the main script. if I compile it and then import the module in my main class, it works fine, but once imported, I can't compile it anymore. (the file is for testing, no x or y values get stored)

def build(function):
    # code for .pyx file
    code = """cpdef long double computeIt(long double count):
    cdef long double i = 0
    cdef long double result = 0
    for i from 0 <= i < count:
        result = _compute(i)
    return result

cpdef long double compute(long double x):
    return _compute(x)
cdef long double _compute(long double x):
    return """ + function
    # end of code

    with open("evalTest.pyx", "w+") as file:
        file.truncate()
        file.write(code)

    os.system("python setup.py build_ext --inplace") # run setup file

setup.py file to compile .pyx:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("evalTest.pyx")
)

Also I tried to compile it to a temp file and replace the .pyd file with an atomic operation but then i get a "Permission denied" error (also if I run the script as admin).

If i first import the cython module and then try to recompile it, I get:

LINK : fatal error LNK1104: cannot open file 'C:\Users\Alexander\Desktop\onTheFlyCythonCompile\evalTest.cp36-win_amd64.pyd'

error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX86\x64\link.exe' failed with exit status 1104

I think the interpreter holds the module and I can't access it but I don't know if thats right. I also tried to unlink the file.

Any Suggestions to solve this problem?

Answer 1

You are correct the the interpreter is holding on to the pyd. To my knowledge it is effectively impossible to unload a c-extension module in a running python interpreter.

Probably the best way to handle this is a randomized / hashed / some other kind of unique filename. This is what for instance the %%cython magic in jupyter does.

In [94]: import cython

In [95]: %load_ext cython

In [96]: %%cython
    ...: def f(a, b):
    ...:     return a + b
# generates ~/.ipython/cython/_cython_magic_5a675eece0e27eef1233e0a3e4d811e5.cp36-win_amd64.pyd

Might also look at numba - it's certainly more convenient for on-the-fly generation.