I need to store a C struct inside a Python object for use in other parts boosted with Cython. I believe a PyCapsule is best suited for this purpose, but the results of my code is not what I am expecting. While the pointer address is correctly returned, the memory appears to have been freed.
I am new to Cython and I am learning to use it to speed up part of my code. For the purpose of asking the question, I have simplified my code and an int is used instead of a struct.
I wrote CythonTest.pyx according to my understanding of the PyCapsule documentation and compiled it with setup.py using the standard command:
python setup.py build_ext --inplace
CythonTest.pyx
#cython: language_level=3
from cpython.pycapsule cimport PyCapsule_New, PyCapsule_IsValid, PyCapsule_GetPointer
class Test:
def __init__(self):
cdef int test = 10
cdef const char *name = "test"
self.vars = PyCapsule_New(<void *>&test, name, NULL)
# Print pointer address
print("{0:x}".format(<unsigned long long>test))
def peek(self):
cdef const char *name = "test"
if not PyCapsule_IsValid(self.vars, name):
raise ValueError("invalid pointer to parameters")
cdef int *test = <int *>PyCapsule_GetPointer(self.vars, name)
print(test[0])
# Print pointer address
print("{0:x}".format(<unsigned long long>test))
setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("CythonTest.pyx"))
Then, I run this with the following Python script.
from CythonTest import Test
test = Test()
print(test.vars)
test.peek()
The console prints out the following:
cbde7ebe70
<capsule object "test" at 0x0000027516467930>
0
cbde7ebe70
It appears that the pointer has been successfully stored in a PyCapsule and retrieved as indicated by the identical address. However, 0 is now stored inside the address instead of 10. I am aware that using an int may have caused it to be garbage collected and changed the nature of the problem, but the same issue is observed when using PyMem_Malloc as well.
So the question is: what is the correct way of using PyCapsule?
test is a local variable (in C) so does not exist beyond the end of the __init__ function, and therefore when you try to access it again in peek the memory has already been used for something else.
You can allocate memory for test of the heap instead, so that the variable persists as long as you need (you need to create a destructor to deallocate it though).
from libc.stdlib cimport malloc, free
# destructor
cdef void free_ptr(object cap):
# This should probably have some error checking in
# or at very least clear any errors raised once it's done
free(PyCapsule_GetPointer(cap,PyCapsule_GetName(cap)))
class Test:
def __init__(self):
cdef int* test = malloc(sizeof(int))
test[0] = 10
cdef const char *name = "test"
self.vars = PyCapsule_New(<void *>test, name, &free_ptr)
# etc