I've been looking all over StackOverflow for this, but can't find a satisfactory answer.
When running kubectl top nodes <node name> I get a memory utilisation of approx. 69% (Kubernetes showing roughly 21Gi of 32Gi being used). But if I go into the system itself and run the free command, as well as the top command, I see a total of 6GB of used memory (i.e. 20% - this is the information under the used column in the output of free) - way less than 69% of the total system memory of 32GB.
Even accounting for the differences in Gi and GB, there's still more than 40% difference unaccounted for. I know that Kubernetes uses the stats reported by /sys/fs/cgroup/memory/memory.usage_in_bytes to report on memory utilisation, but why would this be different than the utilisation reported by other processes on the system (especially sometimes higher)? Which one should I take as the source of truth?
Found answer to my question here: https://serverfault.com/questions/902009/the-memory-usage-reported-in-cgroup-differs-from-the-free-command. In summary, it seems that Kubernetes uses the cgroup memory utilisation, which is reported in /sys/fs/cgroup/memory/memory.usage_in_bytes. The cgroup memory utilisation calculates not only the currently used memory in RAM, but also the "cached" memory (i.e. any memory no longer required by apps that are subsequently free to be reclaimed by the OS, but hasn't been reclaimed yet). The Linux system commands see "cached" memory as "free" but Kubernetes does not (not sure why).