Why can we allocate a 1 PB (10^15) array and get access to the last element, but can't free it?

Question

As known: http://linux.die.net/man/3/malloc

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer.

And we can successfully allocate 1 Petabyte of VMA (virtual memory area) by using malloc(petabyte);: http://ideone.com/1yskmB

#include <stdio.h>
#include <stdlib.h>

int main(void) {

    long long int petabyte = 1024LL * 1024LL * 1024LL * 1024LL * 1024LL;    // 2^50
    printf("petabyte %lld \n", petabyte);

    volatile char *ptr = (volatile char *)malloc(petabyte);
    printf("malloc() - success, ptr = %p \n", ptr);

    ptr[petabyte - 1LL] = 10;
    printf("ptr[petabyte - 1] = 10; - success \n");

    printf("ptr[petabyte - 1] = %d \n", (int)(ptr[petabyte - 1LL]));

    free((void*)ptr);   // why the error is here?
    //printf("free() - success \n");

    return 0;
}

Result:

Error   time: 0 memory: 2292 signal:6
petabyte 1125899906842624 
malloc() - success, ptr = 0x823e008 
ptr[petabyte - 1] = 10; - success 
ptr[petabyte - 1] = 10 

And we can successfully get access (store/load) to the last member of petabyte, but why do we get an error on free((void*)ptr);?

Note: https://en.wikipedia.org/wiki/Petabyte

• 1000^5 PB petabyte
• 1024^5 PiB pebibyte - I use it

So really if we want to allocate more than RAM + swap and to work around overcommit_memory limit, then we can allocate memory by using VirtualAllocEx() on Windows, or mmap() on Linux, for example:

• 16 TiB (16 * 2^40 bytes) then we can use example from Nominal Animal's answer: https://stackoverflow.com/a/38574719/1558037
• 127 TiB (127 * 2^40 bytes) then we can use mmap() with flags MAP_NORESERVE | MAP_PRIVATE | MAP_ANONYMOUS and fd=-1: http://coliru.stacked-crooked.com/a/c69ce8ad7fbe4560

Answer 1

I believe that your problem is that malloc() does not take a long long int as its argument. It takes a size_t.

After changing your code to define petabyte as a size_t your program no longer returns a pointer from malloc. It fails instead.

I think that your array access setting petabyte-1 to 10 is writing far, far outside of the array malloc returned. That's the crash.

Always use the correct data types when calling functions.

Use this code to see what's going on:

long long int petabyte = 1024LL * 1024LL * 1024LL * 1024LL * 1024LL;
size_t ptest = petabyte;
printf("petabyte %lld %lu\n", petabyte, ptest);

If I compile in 64 bit mode it fails to malloc 1 petabyte. If I compile in 32 bit mode it mallocs 0 bytes, successfully, then attempts to write outside its array and segfaults.