I have a call to calloc for 1 element of just over 1 gigabyte. This call returns NULL, and checking errno reveals an insufficient memory error. However, during testing I have almost 4 gigabytes of free RAM, not to mention available virtual memory.
After researching the issue, the only thing I can come up with for why calloc fails is that I do not have enough contiguous memory available. However, it seems to me that if I have 4Gb available memory, it should be easy enough to have ~= 1Gb contiguous memory. Is there a way to check for contiguous memory availability? If this is the problem, is there a straightforward way to "defragment" the memory?
Or does this have something to do with the block size of allocated memory, so that it is actually trying to allocate way more memory than I have available? What are my options for determining cause of failure? What are my options for successfully allocating this much memory?
I am on a native Windows system with 12Gb of memory. Next I will try running in a VM, and giving the VM several gigabytes of memory, and checking if the virtual machine can non-transparently access enough contiguous memory. I will post the results here if I complete that today.
One important key to making that succeed is to have 1GB of virtual address space available to your application. With 32-bit Windows applications, the default situation is that an application starts out with essentially 2GB of address space.
If you do very many allocations in your application prior to the one that is failing, it is quite possible that you have carved out enough of the original address space so as to not leave 1GB of continuous address space left. An application could perform just two small "strategically placed" allocations that would break up the address space such that the remaining portions are all less than 1GB.