Search code examples
pythonnumpymemorymemory-managementram

Loading .npy file takes over 30x longer if RAM usage is high

Here is my RAM usage when loading a .npy file, deleting it with del, calling gc.collect, then loading it again. It behaves as expected and takes about 2.5 seconds.
enter image description here

Here is the same thing, but while I have other things loaded in memory so my total RAM usage is high.
It takes over 90 seconds to load and my usage does this zig zag pattern.
enter image description here

What is going on here? When I delete the array, garbage collect it, then load it again I expect the newly loaded array to just fit right in where the old one was, regardless of my total RAM usage. When it does this zig zag loading it takes way too long.

Solution

  • This is caused by an OS feature called memory paging.

    In short, when the system runs out of memory, it has a mechanism to allocate additional free memory by moving infrequently accessed memory to a file.

    However, this process can take a relatively long time because it involves writing to disk. See how the zigzag climbs instantly and descends slowly? This is because the bottleneck for climbing is access to physical memory, and the bottleneck for descending is access to disk.