Turn off list reflection in Numba

Question

I'm trying to accelerate my code using Numba. One of the arguments I'm passing into the function is a mutable list of lists. When I try changing one of the sublists, I get this error:

Failed in nopython mode pipeline (step: nopython mode backend) cannot reflect element of reflected container: reflected list(reflected list(int64))

I don't actually care about reflecting changes I make to the native list into the original Python list. How do I go about telling Numba not to reflect the changes? The documentation is pretty vague regarding list reflection in Numba.

Thanks,

Answer 1

Quoting directly from the docs:

In nopython mode, Numba does not operate on Python objects. list are compiled into an internal representation. Any list arguments must be converted into this representation on the way in to nopython mode and their contained elements must be restored in the original Python objects via a process called reflection.

Reflection is required to maintain the same semantics as found in regular Python code. However, the reflection process can be expensive for large lists and it is not supported for lists that contain reflected data types. Users cannot use list-of-list as an argument because of this limitation.

Your best bet would be to give a 2D numpy array of shape len(ll) x max(len(x) for x in ll), ll being the list of lists. I myself use something like this to achieve this, and then pass the arr, lengths to the njit compiled function:

def make_2D_array(lis):
    """Funciton to get 2D array from a list of lists
    """
    n = len(lis)
    lengths = np.array([len(x) for x in lis])
    max_len = np.max(lengths)
    arr = np.zeros((n, max_len))

    for i in range(n):
        arr[i, :lengths[i]] = lis[i]
    return arr, lengths

HTH.