I'm populating a list of string shared pointers. At some point in my program, I clear the list. But the memory consumption of my program does not reduce even after I call clear() function of list. Any Idea why?
#include <memory>
#include <string>
#include <list>
#include <iostream>
int main()
{
std::string text = " \
PREFACE \
\
Most of the adventures recorded in this book really occurred; one or two \
were experiences of my own, the rest those of boys who were schoolmates \
of mine. Huck Finn is drawn from life; Tom Sawyer also, but not from an \
individual--he is a combination of the characteristics of three boys whom \
I knew, and therefore belongs to the composite order of architecture. \
\
The odd superstitions touched upon were all prevalent among children and \
slaves in the West at the period of this story--that is to say, thirty or \
forty years ago. \
\
Although my book is intended mainly for the entertainment of boys and \
girls, I hope it will not be shunned by men and women on that account, \
for part of my plan has been to try to pleasantly remind adults of what \
they once were themselves, and of how they felt and thought and talked, \
and what queer enterprises they sometimes engaged in. \
";
std::list<std::shared_ptr<std::string>> data;
for (auto i = 0u; i < 999999; ++i) {
data.push_back(std::make_shared<std::string>(text));
}
std::cout << "Data loaded. Press any key to continue...";
std::cin.get();
data.clear(); // memory does not reduce
std::cout << "Data unloaded. Press any key to continue...";
std::cin.get();
std::cout << "Container size:" << data.size() << std::endl;
std::cout << "Press any key to exit...";
std::cin.get();
return 0;
}
I'm debugging on WSL. I used both linux top (on WSL) and Windows Task Manager to check memory usage at each stop. But valgrind does not report any memory leaks.
P.S. Please don't ask why I'm using shared pointers. Proposing different approach would be useful but the main purpose of this question is to understand this behavior. Since even cppref doesn't explain this. Appreciate if someone can explain this behavior rather than fixing it.
I don't need a fix. I need an explanation.
I used both linux top (on WSL) and Windows Task Manager to check memory usage at each stop. But valgrind does not report any memory leaks.
TLDR: Allocating memory is an expensive operation relative to other code paths. To improve performance, almost all heap managers will continue to hold onto the memory for subsequent allocations instead of giving it directly back from where it came from.
When memory is allocated by your program code, it goes through a tier of memory heaps to grant that allocation. When your program invokes "new" (via make_shared), it calls into the C/C++ runtime to allocate memory. The C/C++ runtime, if it doesn't have a sufficiently large enough contiguous byte range in its heap to grant that allocation, it will thunk down to the process heap via OS specific library calls to ask for more memory. The process heap, if it doesn't have enough to allocate right away, it makes a system call to allocate more virtual memory... and probably a few more heaps as well. And let's not forget that memory likely has to be paged in, but I digress.
Each one of these heap accesses requires taking a lock on a data structure to manage the heap allocation being requested, and possibly a system call to the OS. And possibly some extra effort to collapse or re-arrange blocks of memory as needed. That's why memory heaps are tiered. If every new and delete call were to go directly to the virtual memory manager, program and system performance would be really slow from the sheer number of system calls to do this.
Similarly releasing memory back to where it came from is also a similar performance hit. It's possible that these heaps will release back to its parent heap when it wants to compact, but don't expect it to do that from a single invocation of "delete".
Tools like Top and Task Manager can only observe the amount of virtual memory allocated by the process from the OS. They aren't aware of free'd allocations being managed by the runtime libraries of your program. Whereas Valgrind instruments itself into your code and can hook itself closer to the C++ memory manager. But even Valgrind will report false positives every now and then.