Search code examples
androidperformancecsvgarbage-collectioninstantiation

Android Read Data Into Memory

I'm attempting to read some tabular data into memory for use throughout the running of my application, but instantiating the values repeatedly triggers garbage collection and causes the data to take a long time to load. I've read that this will happen when instantiating large numbers of objects, but I'm having trouble coming up with a way to store the data instead.

The data looks like this, with 12,500 rows and 101 columns:

+-------+-------------+-------------+-----+---------------+
| id    | parameter 1 | parameter 2 | ... | parameter 100 |
+-------+-------------+-------------+-----+---------------+
| 1     | 237.1238220 | 483.2398201 | ... | 403.302910100 |
| 2     | (Float Val) | (Float Val) | ... |  (Float Val)  |
.
.
.
| 12500 | (Float Val) | (Float Val) | ... |  (Float Val)  |
+-------+-------------+-------------+-----+---------------+

Currently, I'm using a BufferedReader to read the data line by line. This doesn't take much time. Then, I split each line on commas using a single character delimiter (",") and store the String[] in an object.

Previously, I was attempting to map the String[] into a Float[], but that took far longer. Using the Android Studio memory profiler, I am able to see that the split is causing the instantiation of a large number of Strings, and that is what is causing the GC events.

I've thought about storing the data in SQLite or Firebase, or setting up a quick server to handle the large data, but all of that seems excessive for pulling 8 MB of data into memory.

Although pulling some data into memory seems like it would be a pretty common task, I'm having trouble finding many other people with the same problem. Am I completely off base with how I'm going about this? Is there some other way to pull this data into memory that I just haven't found?

Any help would be appreciated. Thanks!

Solution

  • 12.500 rows x 101 columns = 1.262.500 strings. That should explain it all. You are creating over 1 million object strings. In terms of memory besides the string value, you must consider that each value will be held as a String object, and pushing to memory over 1 million objects will of course have a large impact. For your scenario, the way to go is an SQLite database. Contrary to what you think it isn't too much. Setting up an SQLite database is simple and very effective for such an amount of data.