Why doesn't the NumPy-C api warn me about failed allocations?

Question

I've been writing a Python extension that writes into a NumPy array from C. During testing, I noticed that certain very large arrays would generate a segfault when I tried to access some of their elements.

Specifically, the last line of the following code segment fails with a segfault:

    // Size of buffer we will write to
    npy_intp buffer_len_alt = BUFFER_LENGTH;

    //
    PyArray_Descr * dtype;
    dtype = PyArray_DescrFromType(NPY_BYTE);
    PyObject* column = PyArray_Zeros(1, &buffer_len_alt, dtype, 0);

    //Check that array creation succeeds
    if (column == NULL){
        // This exit point is not reached, so it looks like everything is OK
        return (PyObject *) NULL;
    }

    // Get the array's internal buffer so we can write to it
    output_buffer = PyArray_BYTES((PyArrayObject *)column);

    // Try writing to the buffer
    output_buffer[0] = 'x'; //No segfault
    output_buffer[((int) buffer_len_alt) - 1] = 'x'; // Segfault here

I checked and found that the error occurs only when I try to allocate an array of about 3GB (i.e. BUFFER_LENGTH is about 3*2^30). It's not surprising that an allocation of this size would fail, even if Python is using it's custom allocator. What really concerns me is that NumPy did not raise an error or otherwise indicate that the array creation did not go as planned.

I have already tried checking PyArray_ISCONTIGUOUS on the returned array, and using PyArray_GETCONTIGUOUS to ensure it is a single memory segment, but the segfault would still occur. NPY_ARRAY_DEFAULT creates contiguous arrays, so this shouldn't be necessary anyways.

Is there some error flag I should be checking? How can I detect/prevent this situation in the future? Setting BUFFER_LENGTH to a smaller value obviously works, but this value is determined at runtime and I would like to know the exact bounds.

---

EDIT:

As @DavidW pointed out, the error stems from casting buffer_len_alt to an int, since npy_intp can be a 64-bit number. Replacing the cast to int with a cast to 'unsigned long' fixes the problem for me.

Answer 1

The issue (diagnosed in the comments) was actually with the array lookup rather than the allocation of the array. Your code contained the line

output_buffer[((int) buffer_len_alt) - 1] = 'x'

When buffer_len_alt (approx value 3000000000) was cast to an (32 bit) int (maximum value 2147483647) you ended up with an invalid address, probably a large negative number.

The solution is just to use

output_buffer[buffer_len_alt - 1] = 'x'

(i.e. I don't see why you should need a cast at all).