Best library for memory-saving storage of large arrays of char/byte/int?

Question

Before re-inventing any wheels that may have been invented many times already, I am looking for a library that would allow me to deal with potentially huge arrays of char (not Char) while keeping heap overhead and unnecessary heap allocation small.

This means the array implementation should allow for

1. access an element as collection.get(long index)
2. store an alement as collection.put(long index, char what) and automatically allocating resizing the array if necessary
3. re-sizing the array by constant-size chunks of some size I would like to choose, e.g. 2^14 elements.

Point 3 is what would be important for my purposes: many implementation simply resize the collection by allocating something that is double to current size, copy over and throw away the old one. This is bad if heap if the collection gets very large and the next resize operation would require all or more than is still left on the heap, although there could still be more than enough on the heap.

Also, the index type should be long so that more than 2^(32-1) elements can be stored in the array.

So if I would implement this, I would probably use a dynamic array of arrays of choosable block size. The first-level array would get resized in any old way (it would not contain too many elements) while the second level array would always be of some fixed block size of 2^N.

Is anyone aware of a library that does this or something similarly memory-efficient?

Answer 1

I think I found the library that fits my needs: The Fastutil library at http://fastutil.di.unimi.it/ has "BigList" implementations for char, int etc. that seem to do exactly what I want in the way I wanted it, and more. Javadoc: http://fastutil.di.unimi.it/docs/