Why pages reclaims still present after using `munmap()`?

Question

for a study project I have to code a reimplementation of malloc() and free() using mmap() and munmap().

I'm running on the last Ubuntu. For my tests I use the command time -v (from /usr/bin/time) which shows me a lot of information about my program including the memory. Here are some examples:

• time -v without using malloc and free

• time -v using only malloc

• time -v using malloc and free (libc)

So we can see Minor page faults which corresponds to the number of reclaimed pages change according to our usage, but especially that if we use free() after a malloc() the number of reclaimed pages return to their initial number which is not the case with my reimplementation:

• time -v using my malloc and free

Here are bits of my code to visualize what I do.

In my malloc():

static t_page *__alloc_page(size_t size)
{
    struct rlimit limit;
    t_page *page;

    getrlimit(RLIMIT_AS, &limit);
    if (size > limit.rlim_max)
        return (NULL);
    page = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (page == MAP_FAILED)
        return (NULL);
    ft_bzero(page, sizeof(t_page));
    page->size = size;
    page->used_size = sizeof(t_page);
    return (page);
}

In my free():

static void __free_page(t_page *page)
{
    t_binding *binder = __get_binder(page);

    binder->count--;
    if (binder->pages == page)
        binder->pages = page->next;
    
    if (page->prev != NULL)
        page->prev->next = page->next;
    if (page->next != NULL)
        page->next->prev = page->prev;

    if (munmap(page, page->size) == -1)
        ft_putstr("free(): munmap error\n");
}

For information my size is always a multiple of getpagesize() (N * getpagesize()).

Here's how I do my tests

First I compile my files malloc.c free.c etc. into a dynamic library (libmalloc.so).
Then I build two binary with the main that follows. One is compiled with my malloc and the other one with the libc.

clang main.c -o libc_malloc
clang main.c -D LIBMALLOC libmalloc.so -o my_malloc

#ifdef LIBMALLOC
# include "../includes/malloc.h"
#else
# include <stdlib.h>
#endif

int main(void)
{
    int i;
    char *addr;

    i = 0;
    while (i < 1024) 
    {
        addr = (char*)malloc(1024);
        addr[0] = 42;
        free(addr);
        i++; 
    }
    return (0);
}

I also have a script that allows me to run my binary with my dynamic library named run.sh:

#!/bin/sh
export LD_LIBRARY_PATH="."
export LD_PRELOAD="`pwd`/libmalloc.so"
$@

Finally I run my two binary with time -v like this:

/usr/bin/time -v ./libc_malloc
./run.sh /usr/bin/time -v ./my_malloc

How to reproduce in a minimalist way

#include <sys/mman.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
    int i;
    char *addr;

    i = 0;
    #ifdef _MMAP_
        printf("mmap\n");
    #else
        printf("malloc\n");
    #endif
    while (i < 1024) 
    {
        #ifdef _MMAP_
            addr = mmap(NULL, 4 * getpagesize(), PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        #else
            addr = malloc(4 * getpagesize());
        #endif
        addr[0] = 42;
        #ifdef _MMAP_
            munmap(addr, 4 * getpagesize());
        #else
            free(addr);
        #endif
        i++; 
    }
    return (0);
}

Copy this main above into a file (main.c).
Create two binary as follows:

clang main.c -o using_malloc
clang -D _MMAP_ main.c -o using_mmap

Then run them with time -v:

/usr/bin/time -v ./using_malloc
/usr/bin/time -v ./using_mmap

What I have tried

While searching the internet I came across this post which has exactly the same problem as mine:
higher page reclaims when using munmap
But the proposed solutions do not work (and i can't use it).
I am not allowed to use functions like posix_madvise() or msync() either...
I tried them anyway to see if they would solve my problem but without success.
I also ran someone else's project. His works well, while we seem to be doing the same thing.
Am I missing something?

Answer 1

I have found my problem. In my main, I continuously execute malloc() and free() one after the other. When the system executes munmap(), it tries to optimize by not immediately removing the page allocated by mmap() in order to reuse it later. However, this still results in the creation of new pages during subsequent calls to mmap().

To fix this, free() need to leaves one remaining page so that munmap() is not called in a loop.

My explanations are not very clear, please feel free to provide more details.