Search code examples
linuxmemory-managementlinux-kerneloperating-systemcgroups

Linux kernel - What is the relationship between page cache, struct address_space, and memory cgroups?


I'm trying to understand the Linux page cache and how it relates to memory cgroups (v2). I know that with cgroupsv1, memory cgroups can be isolated and have independent LRU lists (I assume cgroupsv2 is the same). This and the fact that mm/vmscan.c has many references to mem_cgroups and has a function called shrink_node_memcgs, makes me think that each cgroup has its own page cache. Is this assumption true? Do all pages in a page cache belong to a single cgroup?

If it is true, I know that a page cache is represented by struct address_space (here). How can you find out what cgroup is associated with a given struct address_space? Would I have to just find the first page in the struct address_space and then find the cgroup from the page?


Solution

  • After doing a lot of investigating, there are several issues in my original understanding.

    First, the page cache is a collection of non-contiguous pages residing in memory. It's easy to fall into the trap of thinking about "the page cache" as a single contiguous blocks/pages of memory (similar to a hardware cache), but the page cache is just the collection of some number of pages in memory that is available for future accesses.

    These pages in "the page cache" aren't even stored together virtually. As in, the kernel doesn't keep all pages in the page cache in one global struct or list. Instead, the better way to think about it is that the page cache is actually the union of all LRU lists in the kernel.

    This and the fact that mm/vmscan.c has many references to mem_cgroups and has a function called shrink_node_memcgs, makes me think that each cgroup has its own page cache.

    As mentioned, each cgroup has its own LRU lists. Each cgroup, however, does not have its own page cache. But if you collate all LRUs from all cgroups, you would hold all the pages of the page cache.

    I know that a page cache is represented by struct address_space

    This is incorrect. A struct address_space does represent some number of pages in the page cache, but it isn't itself representative of the entire page cache. struct address_space actually represents cached pages from a single inode (file) or block device (see the host member in the address_space struct). In fact, one of the links in the original post has this quote: "A better name [for struct address_space] is perhaps page_cache_entity or physical_pages_of_a_file." (pg.327, chapter 16)

    How can you find out what cgroup is associated with a given struct address_space?

    Since an address_space corresponds to a single inode (file) and not to a single cgroup, it isn't the case that all pages in this struct are charged to the same cgroup and therefore reside on the same cgroup LRU lists. For instance, imagine that one process in cgroup 1 reads the beginning of a large file, but another process in cgroup 2 reads the end of that file. The pages each process accessed are from the same inode and struct address_space, but some of these pages will be in cgroup 1's LRU lists and others in cgroup 2's.

    So it is not possible to find a cgroup from a struct address_space. Instead, in theory, you could iterate through the struct address_space pages and then find the cgroup that corresponds to each individual page (page->mem_cgroup->css.cgroup).

    Keep in mind that it is possible that a page charged to one cgroup could still be shared/accessed by another process in a different cgroup. See the rules about charging cgroups for shared memory here for v1 (Section 2.3) and here for v2 ("Memory Ownership").

    Addendum:

    During my research, I encountered this article that contributed to my confusion and led me to think an address_space was associated with a single cgroup. Image 4.2 makes it seem like address_space is buried within an mm_struct; and since mm_struct is specific to a process, then this address_space should also correspond to a process and, by extension, that process' cgroup. In reality, the process that this mm_struct corresponds to holds a file descriptor (represented by struct file) of a file and this file descriptor leads to the file inode and its corresponding address_space. This address_space is needed to find pages specifically from this file in the "page cache".