ESP8266 Fails to add char to a very long String (>8000 chars)

Question

After correctly getting the payload from an HTTPS request, adding the client's chars to a String stops after about 8000 characters, then resumes and stops again a few times Here's a snippet of my code:

long streamSize = 0;
Serial.println("Now reading payload...");
while (stream.connected()) {
    while (stream.available() > 0) {
        char ch = (char)stream.read();
        Serial.println((String)"Reading [" + ++streamSize + "] " + ch);
        ret += ch;
        Serial.println(ret.length());
    }
}

Which works fine, until:

Reading [8685] t
8685
Reading [8686] r
8686
Reading [8687] u
8687
Reading [8688] m
8687
Reading [8689]  
8687
Reading [8690] e
8687
[Resumes correctly appending chars]
Reading [9226] i
8748
Reading [9227] p
8749
Reading [9228] t
8750
Reading [9229] i
8751
Reading [9230] o
8751
Reading [9231] n
8751

And so on for quite a few times. Memory Heap size doesn't seem to be the issue, as I'm getting 14128 free bytes from system_get_free_heap_size() after appending everything. I'm using a Wemos D1 R1, and this is the file I'm trying to fully read, for testing, using the Github API

Answer 1

I've found out that Arduino can fail to concatenate strings because of a low amount of free memory. Moreover, String class in Arduino seems not to have an error handler, so it can silently fail when - for example - memory is too fragmented.

See here: from Arduino forum and here: from a discussion in Stack Overflow

In many cases they suggest that you could pre-allocate the buffer with a String reserve(int) call.

Maybe you could not know in advance how big your string will grow, but maybe you could managing it. For example, by calling two times your https target. First time just to know how big your answer will be (and so you will able to allocate the exact amount of memory); second time to effectively read.