Working with larger that 2 billion elements big-arrays in Java

Question

I am constantly having to work with large arrays in Java, constantly testing the ~2B elements limitation.

Is there a convenient way to work with arrays of larger size? The list of operations I need:

• create a new array;
• set/get i-th element;
• extend the array by allocating a new one of larger size and copying contents of the old array;
• copying contents of another array (rather small << 2B) to the array.

P.S. It seems that this problem has been thinked over by a number of engineers. Nice article with references could be found at: https://www.nayuki.io/page/large-arrays-proposal-for-java

Answer 1

I recommend fastutil library. I use it mainly for the space-efficient Java collections (using primitive arrays internally), but there's also a section about Big data structures for collections with > 2^31 elements.