Search code examples
c++arraysvectorallocation

Declaring and initializing very large arrays on the heap


I have an array of about 66,000 elements, with each element being a POD struct of integral data types. This array is constant and will never change, so my original thought was to just put it in as a constant global variable.

I declared it in a header as extern and initialized it in a cpp file, like (obviously simplified here):

const PODStruct bigArray[] = 
{
    {1,2,3,4} , {1,2,3,5} , ....
}

with some editing in a text editor so it wasn't just one continuous line.

--EDIT: I was reminded that global variables are of course not stack-allocated, so the paragraph that was here went away! However, what if I still would rather have the data in a vector?

I was thinking since C++11 allows that same syntax for std::vector initialization, I could just use that with a simple edit and have a vector instead. However, in MSVC++ 2013, when I attempt to compile it says I've hit compiler limits. I looked through the C++ standards for compiler limits and MSVC++13's deviations from it, but nothing seemed to be directly the cause. I'm guessing it has to do with how that initializer-list syntax is actually implemented.

I can get the array itself into a vector using the constructor in the answer here: How to initialize std::vector from C-style array?

However, then I'd still have the array in memory twice, right? It's not on the stack like I originally feared and not that big, so it's not a huge deal, but seems like a sloppy solution.

I'm thinking I could create a class with a default constructor, and declare and initialize the typed-out table in there. Then I can declare the vector in the constructor and construct it with the array. The class's only member could just be the vector.

I could just declare that class then create a global instance of it, right? That'd be similar to the behavior I had with the global array. If I wanted to get away from that, is the best approach to declare the class first thing under main, and then pass it around to the functions and methods that need the table?

Should I want to get away from that? This data is, despite a lot of it, along the lines of PI = 3.4.


Solution

  • Your idea about storing your "huge constant array" in a compile-size generated constant is ok, that what I'd do.

    If you try to move all this to a vector or other variant of heap-allocated array, then you'd simply duplicate the data, since the initialization data resides in your executable image anyway.

    To workaround the (idiotic) MSVC 2013 compiler limit that's what I'd try.

    • Switch to MSVC 2010 compiler. See the build options for your .cpp file, in MSVC 2013 you may set the "platform toolset" of MSVC 2010.
    • Try to redefine your data type. For instance, instead of having an array of structs, try an array of (constant) pointers to structs. All this should be compile-time generated as well.

    With some efforts most probably you may work this around. Good luck.