Search code examples
clinuxmmap

mmap call works on one system but fails with "Cannot allocate memory" on another. Both running same linux image, with plenty of memory

I'm trying to run some code I didn't write, that needs a large chunk (~1GB) of contiguous memory. I'm trying it on two different hardware configurations, but with the same linux binary. It runs on one system but gives an error "Cannot allocate memory" on the other.

uint64_t alloc_flags = MAP_PRIVATE | MAP_POPULATE | MAP_ANONYMOUS | MAP_HUGETLB | (30 << MAP_HUGE_SHIFT) 

mem->buffer = (char *)mmap(NULL, mem->size, PROT_READ | PROT_WRITE,
                  alloc_flags, mem->fd, 0);
       if (mem->buffer == MAP_FAILED) {
       perror("[ERROR] - mmap() failed with");
       exit(1);
   }

Any ideas of what might be the problem, or what to look at?

/proc/meminfo looks about the same on both systems.

I tried without success: echo 20 > /proc/sys/vm/nr_hugepages

EDIT: both systems /sys/kernel/mm/hugepages/ have: "hugepages-1048576kB hugepages-2048kB"

.. /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages reveals 1 on the successful system, and 0 on the failing system!

Solution

  • Having plenty of memory is not enough. Hardware requires each page be contiguous range of physical memory aligned by page size.

    If your systems memory is fragmented and doesn't have enough contiguous regions, the allocation will fail.

    You can ask kernel to reserve some amount of memory at boot time through kernel parameters:

    hugepagesz=1G hugepages=2

    This should allocate 2 1GB pages. Note that your CPU needs to have pdpe1gb flag set:

    grep pdpe1gb /proc/cpuinfo

    This should not be a problem since all CPUs since Westmere (released in 2010).