Write data to an SD card through a buffer without a race condition

Question

I am writing firmware for a data logging device. It reads data from sensors at 20 Hz and writes data to an SD card. However, the time to write data to SD card is not consistent (about 200-300 ms). Thus one solution is writing data to a buffer at a consistent rate (using a timer interrupt), and have a second thread that writes data to the SD card when the buffer is full.

Here is my naive implementation:

#define N 64
char buffer[N];
int count;

ISR() {
    if (count < N) {
        char a = analogRead(A0);
        buffer[count] = a;
        count = count + 1;
    }
}

void loop() {
    if (count == N) {
        myFile.open("data.csv", FILE_WRITE);

        int i = 0;
        for (i = 0; i < N; i++) {
            myFile.print(buffer[i]);
        }
        myFile.close();
        count = 0;
    }
}

The code has the following problems:

1. Writing data to the SD card is blocking reading when the buffer is full
2. It might have a race conditions.

What is the best way to solve this problem? Using a circular buffer, or double buffering? How do I ensure that a race condition does not happen?

Answer 1

You have rather answered your own question; you should use either double buffering or a circular buffer. Double-buffering is probably simpler to implement and appropriate for devices such as an SD card for which block-writes are generally more efficient.

Buffer length selection may need some consideration; generally you would make the buffer the same as the SD sector buffer size (typically 512 bytes), but that may not be practical, and with a sample rate as low as 20 sps, optimising SD write performance is perhaps not an issue.

Another consideration is that you need to match your sample rate to the file-system latency by selecting an appropriate buffer size. In this case the 64 sample buffer buffer will fill in a little more than three seconds, but the block write takes only up-to 300 ms - so you could use a much smaller buffer if required - 8 samples would be sufficient - although be careful, you may have observed latency of 300 ms, but it may be larger when specific boundaries are crossed in the physical flash memory - I have seen significant latency on some cards at 1 Mbyte boundaries - moreover card performance varies significantly between sizes and manufacturers.

An adaptation of your implementation with double-buffering is below. I have reduced the buffer length to 32 samples, but with double-buffering the total is unchanged at 64, but the write lag is reduced to 1.6 seconds.

// Double buffer and its management data/constants
static volatile char buffer[2][32];
static const int BUFFLEN = sizeof(buffer[0]);
static const unsigned char EMPTY = 0xff;
static volatile unsigned char inbuffer = 0;
static volatile unsigned char outbuffer = EMPTY;

ISR()
{
    static int count = 0;

    // Write to current buffer
    char a = analogRead(A0);
    buffer[inbuffer][count] = a;
    count++ ;

    // If buffer full...
    if( count >= BUFFLEN )
    {
        // Signal to loop() that data available (not EMPTY)
        outbuffer = inbuffer;

        // Toggle input buffer
        inbuffer = inbuffer == 0 ? 1 : 0;
        count = 0;
    }
}

void loop()
{
    // If buffer available...
    if( outbuffer != EMPTY )
    {
        // Write buffer
        myFile.open("data.csv", FILE_WRITE);
        for( int i = 0; i < BUFFLEN; i++)
        {
            myFile.print(buffer[outbuffer][i]);
        }
        myFile.close();

        // Set the buffer to empty
        outbuffer = EMPTY;
    }
}

Note the use of volatile and unsigned char for the shared data. It is important that data shared between concurrent execution contexts is accessed explicitly and atomically; access to an int on 8-bit AVR based Arduino requires multiple machine instructions and the interrupt may occur part way through a read/write in loop() and cause an incorrect value to be read.