Share portion/area of mmap'ed memory

Question

I have several processes which performed mmap() of a specific size (0x8000). I would like to share only a portion of this memory space between these processes as shown in the following diagram:

      0x0             0x2000-0x3000           0x8000
p1:   [MEM. PRIVATE]  [MEM. SHARING]  [MEM. PRIVATE]
p2:   [MEM. PRIVATE]  [MEM. SHARING]  [MEM. PRIVATE]

In this scenario, the memory allocated by mmap() must be only shared between the range 0x2000-0x3000. Other portions are private (MEM. PRIVATE).

Is there a system call to perform the sharing after the call to mmap()? I tried with shm_open() beforehand but the entire range is shared.

Answer 1

Sounds like you want 3 mappings, with only the middle one being MAP_SHARED.

But instead of 3 separate mmap calls, make two. First a MAP_PRIVATE of the whole length, then a MAP_FIXED|MAP_SHARED of the shared region at offset 0x2000, replacing the middle of the first mapping. The first mapping could be MAP_ANONYMOUS, not backed by the file at all, since you're not sharing it.

If it should be zero-initialized, you don't need a disk file or part of an shm region for that. Being private, writes to it won't persist anywhere, but could start with a non-zero initializer if you write data to the file some other way.

With MAP_FIXED, the first arg to mmap isn't just a hint, it's definitely where your mapping will be, even if that means replacing (part of) a previous mapping at that virtual address as if by munmap.

void *setup_mapping(int fd)
{
   const size_t total_len = 0x8000;
   const size_t shared_mem_offset = 0x2000, shared_len = 0x1000;
   const size_t shared_file_offset = 0;  // the file *only* holds the shared part
  // unless you have some non-zero data for the private mapping to start with

   // Private anonymous non-shared mapping, like malloc.
   // Letting the kernel pick an address
   char *base = mmap(NULL, total_len, PROT_READ | PROT_WRITE, 
                      MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
               // or without MAP_ANONYMOUS, using fd, 0 
   if (base == MAP_FAILED)
       return MAP_FAILED;

  // replace the middle of that mapping with our shared file mapping
   void *shared = mmap(base+shared_mem_offset, shared_len, PROT_READ|PROT_WRITE, 
                      MAP_SHARED|MAP_FIXED, fd, shared_file_offset);

   if (shared == MAP_FAILED){
       munmap(base, total_len);   // if munmap returns error, still just return
       return MAP_FAILED;
   }

   return base;
}

I used char *base so math on it would be well-defined. GNU C does define pointer math on void* as working like on char*.

BTW, a single munmap can tear down the whole region when you're done, regardless of it being 2 or 3 mappings. Or of course just exit the process.

---

Starting with an mmap of the full length of your total region means you don't need to look for a free range of virtual addresses to use as hints; the first mmap will find and claim one so you can just safely MAP_FIXED over part of it. This is safe even if other threads are allocating memory at the same time; they can't randomly pick a conflicting address between two mmap calls like they could if you did 3 separate mappings with just hint addresses or with MAP_FIXED_NOREPLACE. It's unlikely that it would be a problem in practice.

Also, this would only make 2 system calls instead of 3 so it's more efficient. (The internal work of mapping more pages, and of replacing a mapping, should be minor, especially when you haven't touched that memory yet to actually fault it in.)

It's also less code to write, and fewer error return values to check, thus fewer corner cases for partial failure.