Search code examples
c++stdvectorstdstring

Why does this vector throw a bad allocation exception?


Why does this seemingly innocent function throw a bad allocation exception for noUrls=300,000,000?

    #include <string>
    #include <vector>
    #include <algorithm>

    void generateUrls(int noUrls)
    {
        std::vector<std::string> urls;
        urls.resize(noUrls); //this always works
        std::size_t i = 0;
        std::string url = "abcdefghijklmnop";
        
        urls[i] = "https://www." + url + ".com/home/index";
        i++;
    
        while (std::next_permutation(url.begin(), url.end()) && (i < noUrls))
        {
            urls[i] = "https://www." + url + ".com/home/index"; //this where it throws
            i++;
        }
    }

    int main()
    {
        generateUrls(100000000);
        //do something with the result
        return 0;
    }

Why?

Clues:

  • noUrls=300,000,000 is not causing an overflow, my platform's maximum int size of 2,147,483,647
  • The resulting vector will be about 12GB large
  • My system has 8GB physical RAM, but a page file size of another 16GB, so would have expected it to just stick some of the vector in the page file if it runs out of physical RAM. (A stupid assumption?)

Solution

  • You are only counting characters, but std::string is not just characters. On my platform, sizeof(std::string) is 32. That's about 9GiB for an array of zero-length strings, before you start adding any characters.

    If a string is short, most implementations keep the characters inside those 32 bytes to avoid allocations. But your strings are longer than that, so the characters are allocated on the free store. Add 12 GiB worth of characters, and you are well out of memory.