Search code examples
c++memoryvectorboost-unordered

Memory usage in C++ program


I wrote a program that needs to handle a very large data with the following libraries:

  • vector
  • boost::unordered_map
  • boost::unordered_multimap

So, I'm having memory problems (The program uses a LOT) and I was thinking maybe I can replace this libraries (with something that already exists or my own implementations):

So, three questions:

  • How much memory I'd save if I replace vector with a C array? Is it worth it?
  • Can someone explain how is the memory used in boost::unordered_map and boost::unordered_multimap in the current implementation? Like what's stored in order to achieve their performance.
  • Can you recommend me some libraries that outperform boost::unordered_map and boost::unordered_multimap in memory usage (But not something too slow) ?

Solution

  • I ended up by changing boost::unordered_multimap for std::unordered_map of vector.

    boost::unordered_multimap consumes more than two times the memory consumed by std::unordered_map of vector due the extra pointers it keeps (at least one extra pointer per element) and the fact that it stores the key and the value of each element, while unordered_map of vector only stores the key once for a vector that contains all the colliding elements.

    In my particular case, I was trying to store about 4k million integers, consuming about 15 GB of memory in the ideal case. Using the multimap I get to consume more than 40 GB while using the map I use about 15 GB (a little bit more due the pointers and other structures but their size if despicable).