Search code examples
cmemory-managementshared-memorymmap

Reserved Memory Equals Shared Memory but Memory is Never Reserved

I am currently editing a program I inherited to be able to work with 23 GB files. As such, to maintain low memory, I am using mmap to load arrays which I had created in a previous program. However, I load these arrays, and then enter into a function and the shared and reserved memory spikes, even though I do not believe I ever allocate anything. When running, the memory starts at 0 and then quickly increases to 90% (~36GB as I have 40GB of ram) and stays there. Eventually, I start needing memory (less than 30GB) and the program then gets killed.

Usually, I would suspect that this issue would be due to allocation, or that I was somehow allocating memory. However, I am not allocating any memory (although I am reading in mmaped files).

The curious thing is that the memory reserved is equal to the amount of memory shared (see attached screenshot).

Output

The functions that I wrote to access mmaped arrays:

double* loadArrayDouble(ssize_t size, char* backupFile, int *filedestination) {
    *filedestination = open(backupFile, O_RDWR | O_CREAT, 0644);
    if (*filedestination < 0) {
        perror("open failed");
        exit(1);
    }
    // make sure file is big enough
    if (lseek(*filedestination,size*sizeof(double), SEEK_SET) == -1) {
        perror("seek to len failed");
        exit(1);
    }
    
    if (lseek(*filedestination, 0, SEEK_SET) == -1) {
        perror("seek to 0 failed");
        exit(1);
    }

    double *array1 = mmap(NULL, size*sizeof(double), PROT_READ | PROT_WRITE, MAP_SHARED, *filedestination, 0);
    if (array1 == MAP_FAILED) {
        perror("mmap failed");
        exit(1);
    }

    return array1;
}

Please let me know if there is any other code to include.. It appears that the memory increases significantly even though double* file1 = loadArrayDouble(SeqSize, "/home/HonoredTarget/file1", &fileIT); is called multiple times (for each of the 6 arrays)

Solution

  • "Res" is short for "resident", not "reserved". Resident memory refers to the process memory which the kernel happens to have resident at the moment; the virtual memory system might drop a resident page at any moment, so it's not in any way a limitation. However, the kernel attempts not to swap out pages which seem to be active. The OOM killer will act if your process is churning too many pages in and out of memory. If you use data sequentially, then it usually doesn't matter how much you have mmap'ed, because only the recent pages will be resident. But if you skip around in the memory, reading a bit here and writing a bit there, then you'll create more churn. That seems like what is happening.

    "shr" (shared) memory does in fact refer to memory which could be shared with another process (whether or not it actually is shared with another process). The fact that you use MAP_SHARED means that it is not surprising that all of your mmap'ed pages are shared. You need MAP_SHARED if your program modifies the data in the file, which I guess it does.

    The "virt" (virtual) column measures how much of your address spaced you've actually mapped (including memory mapped to anonymous backing storage by whatever dynamic allocation library you're using.) 170G seems a bit high to me. If you have six 23GB files mapped simultaneously, that would be 138GB. But perhaps those numbers were just estimates. Anyway, it doesn't matter that much, as long as you're within the virtual memory limits you've set. (Although page tables do occupy real memory, so there is some effect.)

    Memory mapping does not save you memory, really. When you mmap a file, the contents of the file still need to be read into memory in order for your program to use the data. The big advantage to mmap is that you don't have to futz around with allocating buffers and issuing read calls. Also, there is no need to copy data from the kernel buffer into which the file is read. So it can be a lot easier and more efficient, but not always; it depends a lot on the precise access pattern.

    One thing to note: the following snippet does not do what the comment says it does:

        // make sure file is big enough
    if (lseek(*filedestination,size*sizeof(double), SEEK_SET) == -1) {
        perror("seek to len failed");
        exit(1);
    }

    lseek only sets the file position for the next read or write operation. If the file does not extend to that point, you'll get an EOF indication when you read, or the file will be extended (sparsely) if you write. So there's really not much point. If you want to check the file size, use stat. Or make sure you read at least one byte after doing the seek.

    There's also not a lot of point using O_CREAT in the open call, since if the file doesn't exist and thus gets created, it will have size 0, which is presumably an error. Leaving O_CREAT off means the open call will fail if the file doesn't exist, which is likely what you want.

    Finally, if you are not actually modifying the file contents, don't mmap with PROT_WRITE. PROT_READ pages are a lot easier for the kernel to deal with, because they can just be dropped and read back in later. (For writable pages, the kernel keeps track of the fact that the page has been modified, but if you aren't planning on writing and you don't allow modification, that makes the kernel's task a bit easier.)