Search code examples
concurrencymultiprocessingsystem-design

Multi processes read different part of a big binary file simultanously


I have a large binary file, and it is saved on a NFS share disk. In the cluster, I want multiple processes to simultaneously read this big file. Each process gets a file pointer, opens the big file and reads starting from the supplied pointer and read some size of bytes.

How do I design this project? As far as I concerned, it is similar to some concurrency databases. Is there any lightweight library or open-source projects related to my project? I use the C++ language.


Solution

  • Not sure if there is a point to use a library.

    You could use basic stuff. Open and reposition yourself in the file and then perform the read:

    http://www.cplusplus.com/reference/fstream/ifstream/open/ http://www.cplusplus.com/reference/istream/istream/seekg/

    or

    http://www.cplusplus.com/reference/cstdio/fopen/ http://www.cplusplus.com/reference/cstdio/fseek/