file size and buffer overshoot

Question

I have a function which opens a file from an SD card, uses the file size to set the size of a buffer, writes a block of information to that buffer, then does something with that information, as shown in this code:

char filename = "filename.txt";
uint16_t duration;
uint16_t pixel;
int q = 0;
int w = 0;
bool largefile;
File f;
int readuntil;
long large_buffer;

f = SD.open(filename);

if(f.size() > 3072) {
    w = 3072;
} else {
    w = f.size();
}
uint8_t buffer[w];

while(f.available()) {
    f.read(buffer, sizeof(buffer));

    while(q < sizeof(buffer)) {
        doStuffWithInformation(buffer[q++]);
    }
    q=0;
}
f.close;

This works great with smaller file sizes, but anything over the hard limit buffer size of 3072 (which I arrived at empirically, its just the amount of memory that can be safely committed to this function), runs into a problem. Larger files read fine until they hit the last loop of while(f.available()), where they read the end of the file, but then continue reading the buffer, the tail end of which is filled with data from the last loop, that wasn't overwritten by the latest f.read(). How can I make sure that the last loop of the while(f.available()) function only works with the information that was written to the buffer during the current loop? My only idea right now is to solve for factors of the file size, and set the buffer size as the largest factor less than 3072, but this seems intensive to run every time this function is called. Is there an elegant solution staring me in the face?

Answer 1

Your program is not behaving correctly because f.read() is not guaranteed to read the whole buffer. Moreover, it is bound to happen when you read the last chunk of the file, unless the file size is a factor of buffer size (3072 in your case).

While Arduino specification (https://www.arduino.cc/en/Reference/FileRead) doesn't say so, SD.read function returns the number of bytes read. See code of the library here: https://github.com/arduino-libraries/SD/blob/master/src/utility/SdFile.cpp, int16_t SdFile::read(void* buf, uint16_t nbyte)

Knowing that, you should change your loop as following (while also rewriting it as a for loop for better readability and removing q definition above):

while(f.available()) {
    uint16_t sz = f.read(buffer, sizeof(buffer));

    for (uint16_t q = 0; q < sz; ++q) {
        doStuffWithInformation(buffer[q]);
    }
}

On a side note, now, when you have this logic in place, it would make sense for you to do away with variable length array and use a fixed buffer of size 512 - the standard sector size on the SD card. Most likely, it will yield the same performance in regards to read, and slightly better performance in regards to sizeof, which will becomes a compile-time constant rather than a run-time calculation. This also makes your program simpler. This makes for following code:

f = SD.open(filename);
...
uint8_t buffer[512];