Search code examples
c++boostcross-platformmemory-mapped-filesprefetch

Are there cross-platform ways to promote prefetching for reading a large Boost memory mapped_file?


I have a C++ application for Windows and Linux (e.g. Boost 1.53.2 on Amazon Linux) that is using Boost::iostream::mapped_file (i.e. a memory mapped file). The documentation doesn't mention "prefetch".

The application needs to sequentially read through large read-only files rapidly. Sometimes these files will be larger than available memory. So loading the whole file into memory at once may not be an option. But in all cases, processing will proceed from the beginning to the end sequentially.

It would be helpful if prefetching of upcoming pages happens in a way that keeps ahead of the processing of the pages (i.e. upcoming pages in memory before they are needed), yet not so far ahead that not-yet-processed pages are dropped from memory to make room.

I'm wondering if there are helpful cross-platform ways (Windows and Linux) to give hints or direction or otherwise promote the automated prefetching of the pages that will be needed in the not distant future. I expect the OS might do this to some degree automatically, but I am wondering if there is a convenient technique I should be using to improve on whatever is the default behavior.

Thanks in advance!


Solution

  • Not sure how portable, but I included fadvise and madvise in this answer:

    Seems there are some good pointers for non-POSIX windows here: What is fadvise/madvise equivalent on windows?