Can't download a 100mb file on Android with LibGDX Net

Question

I have a simple code which sends a Http request to my external server to download a .txt file with a size of 100mb. Smaller files, like 40mb are working, but there are some problems with bigger ones. Let me show you some code:

Net.HttpRequest request = new Net.HttpRequest(Net.HttpMethods.GET);
    request.setTimeOut(2500);
    String assetsUrl = "http://111.111.111.111/100mb.txt";
    request.setUrl(assetsUrl);


    // Send the request, listen for the response
    // Asynchronously
    Gdx.net.sendHttpRequest(request, new Net.HttpResponseListener() {
        @Override
        public void handleHttpResponse (Net.HttpResponse httpResponse) {


            InputStream is = httpResponse.getResultAsStream();
            OutputStream os = Gdx.files.local("100mb.txt").write(false);

            byte[] bytes = new byte[1024];
            int count = -1;

            try {

                while ((count = is.read(bytes, 0, bytes.length)) != -1) {
                    os.write(bytes, 0, count);
                }

            } catch (IOException e) {
                e.printStackTrace();
            }


        }

There's some more code to display progress but it's not important here. The problem is that the file's size is 100mb but Android magically allocates 400mb+ RAM while downloading it and comes with the error:

Waiting for a blocking GC Alloc

WaitForGcToComplete blocked for 12.906ms for cause Alloc

Starting a blocking GC Alloc

Starting a blocking GC Alloc

Suspending all threads took: 35.332ms

Alloc partial concurrent mark sweep GC freed 214(21KB) AllocSpace objects, 1(200MB) LOS objects, 6% free, 216MB/232MB, paused 1.076ms total 130.115ms

Suspending all threads took: 205.400ms

Background sticky concurrent mark sweep GC freed 364(10KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 416MB/416MB, paused 10.448ms total 304.325ms

Starting a blocking GC Alloc Starting a blocking GC Alloc

Alloc partial concurrent mark sweep GC freed 113(3KB) AllocSpace objects, 0(0B) LOS objects, 3% free, 416MB/432MB, paused 290us total 17.611ms

Starting a blocking GC Alloc

Alloc sticky concurrent mark sweep GC freed 31(912B) AllocSpace objects, 0(0B) LOS objects, 3% free, 416MB/432MB, paused 268us total 6.474ms

Starting a blocking GC Alloc

Alloc concurrent mark sweep GC freed 43(13KB) AllocSpace objects, 0(0B) LOS objects, 3% free, 415MB/431MB, paused 268us total 15.008ms

Forcing collection of SoftReferences for 300MB allocation

Starting a blocking GC Alloc

Alloc concurrent mark sweep GC freed 42(1256B) AllocSpace objects, 0(0B) LOS objects, 3% free, 415MB/431MB, paused 286us total 12.426ms

Throwing OutOfMemoryError "Failed to allocate a 314572860 byte allocation with 16770608 free bytes and 96MB until OOM"

When I run the downloading process, I can see on my device that I have 1GB allocated (by system) and 600 mb free while the app uses 20-30 mb. After couple seconds, my app starts to allocate more and more memory and I can see, that it uses 400mb+ and the crash comes when it comes to the maximum as you can see in the logs.

Maybe I don't understand it, but shouldn't it only allocate the required 100mb of RAM to store the chunks of data? I'm nearly 100% sure that there is no leak in my app - memory is being consumed by the downloading process (which is for sure called only once).

Answer 1

Nothing you're doing looks weird, so I took a look at LibGDX, particularly getResultAsString; eventually you get down to com.badlogic.gdx.utils.StreamUtils.copyStreamToString

    public static String copyStreamToString (InputStream input, int estimatedSize, String charset) throws IOException {
        InputStreamReader reader = charset == null ? new InputStreamReader(input) : new InputStreamReader(input, charset);
        StringWriter writer = new StringWriter(Math.max(0, estimatedSize));
        char[] buffer = new char[DEFAULT_BUFFER_SIZE];
        int charsRead;
        while ((charsRead = reader.read(buffer)) != -1) {
            writer.write(buffer, 0, charsRead);
        }
        return writer.toString();
    }

This doesn't look terrible, but my guess is something's going on with StringWriter, causing repeated reallocation of an array internally. Looking at this answer seems to confirm my suspicions.

StringWriter writes to a StringBuffer internally. A StringBuffer is basically a wrapper round a char array. That array has a certain capacity. When that capacity is insufficient, StringBuffer will allocate a new larger char array and copy the contents of the previous one. At the end you call toString() on the StringWriter, which will again copy the contents of the char array into the char array of the resulting String.

So, in short you should probably find another method for downloading the file. This question may have some more robust solutions.