Search code examples
linuxwindowswinapimmapmemory-mapped-files

Win64/Linux: Mmap readonly chunks (few MB) of memory at/after the end of a file

Is it possible in Windows and Linux to mmap chunks of memory on a file that isn't as large as the mapping?

The file may contain only a few bytes worth of values, but I want to create multiple 1MB (in that ballpark) memory-mapping and read from this memory as the file grows. The file is written-to by the same program. I will never touch memory beyond the end of the file. But I don't want to redo the mmap calls with every new values that are appended into the file!

  • Linux: I checked man 2 mmap and it doesn't appear to forbid this kind of usage.
  • Windows: I checked the Windows API. The API devides the task into two functions, CreateFileMapping and MapViewOfFile. The latter is similar to mmap, and wants a handle that is returned by the former. But the size passed to MapViewOfFile is restricted to whatever size was given to CreateFileMapping! Can I simply call CreateFileMapping with whatever my max file size is going to be, let's say 100TB and Windows will let me create views of the file as the file grows till 100TB?

I checked folly's MemoryMap implementation, and it won't allow mappings larger than the file size. Neither does python's mmap. I haven't found a definitive answer for boost.interprocess.

Solution

  • Subsequently I found an answer. I would like to quote my comments made to YangXiaoPo-MSFT's answer who was kind enough to provide me with first-hand MSFT information, after which I did some research on my own.

    I subsequently discovered another answer that goes into some detail on how to resize an existing file mapping to be larger. It involves directly calling the Zw / Nt functions and not using the CreateFileMapping wrapper. As far as I know, CreateFileMapping creates a special "section" object that's backed by an arbitrary file - like there are sections .data for example that are too backed by files. Usually these sections are fixed size. But the sepecial Nt functions allow extending the section.

    The mentioned answer about NtExtendSection is at https://stackoverflow.com/a/55068177/34509 .

    I also uttered the following

    Although I will probably not map the entire file into memory - it may grow muiltiple terabytes and even on 64bit, I'm unsure whether windows will find a large enough contiguous VM segment. I will create the file mapping over the entire file, and then map only individual chunks of the file.