Storing unsafe C derivative of temporary Python reference error in Cython

Question

I was writing code to store a (potentially) very large integer value into an array of chars referenced by a pointer. My code looks like this:

cdef class Variable:

    cdef unsigned int Length
    cdef char * Array

    def __cinit__(self, var, length):
        self.Length = length
        self.Array = <char *>malloc(self.Length * sizeof(char))    # Error
        for i in range(self.Length):
            self.Array[i] = <char>(var >> (8 * i))

    def __dealloc__(self):
        self.Array = NULL

When I tried compiling the code, I got the error, "Storing unsafe C derivative of temporary Python reference" at the commented line. My question is this: which temporary Python reference am I deriving in C and storing, and how do I fix it?

Answer 1

The problem is that under the hood a temporary variable is being created to hold the array before the assignment to self.Array and it will not be valid once the method exits.

Note that the documentation advises:

the C-API functions for allocating memory on the Python heap are generally preferred over the low-level C functions above as the memory they provide is actually accounted for in Python’s internal memory management system. They also have special optimisations for smaller memory blocks, which speeds up their allocation by avoiding costly operating system calls.

Accordingly you can write as below, which seems to handle this use case as intended:

from cpython.mem cimport PyMem_Malloc, PyMem_Realloc, PyMem_Free

cdef class Variable:

    cdef unsigned int Length
    cdef char * Array

    def __cinit__(self, var,size_t length):
        self.Length = length
        self.Array = <char *>PyMem_Malloc(length * sizeof(char))
        #as in docs, a good practice
        if not self.Array:
            raise MemoryError()

        for i in range(self.Length):
            self.Array[i] = <char>(var >> (8 * i))

    def __dealloc__(self):
        PyMem_Free(self.Array)