How to avoid damaging SD card for large writes?

Question

Ok, first a little background to help make my question clear:

I am working on a device that collects certain data from sensors and posts them to a server using a GSM modem. As a GSM connection is not 100% reliable, it would contain a logging mechanism that would write unsent data to an SD card.

We are using Chan's FatFs module for providing us with a file system as we want the log to be readable on a PC.

Now I've been testing the FAT system for boundary conditions, i.e., trying to fill up the card completely.

In the first run I opened the file and set the code to keep writing a string until the drive was full. The program would synch after every write.

I left the code running overnight.

The next day, I examined the SD card. I found that the file was only 150 MB in size. There were about 1.2 million lines written to it. The card could still be read from but not written to or formatted.

Next time I tried the same type of test, but this time I used the f_lseek() function to pre-allocate the file to 1GB. It would then write to that file until that limit was reached. This time the data would be synced after 50 writes. It would then close that file and open another to do the same.

As you can guess another brave little card lost it's mind that day.

So these are what I would like help with :

1. How to prevent damage to the card while writing large amounts of data?
2. Does leaving the file open for extended periods have any negative effects?

Since the full code may be too long, here's the main part where the writing happens

for(file_count=3;file_count>=0;--file_count){

    ax_log_msg(E_LOG_INFO,"===================================");

    ax_log_msg(E_LOG_INFO,file_names[file_count]);


    f_open(&file_ptr,file_names[file_count],FA_WRITE|FA_OPEN_ALWAYS);

    if(result!=FR_OK){

        ax_log_msg(E_LOG_INFO,"\n\rf_open Failed\n\rResult code");
        ax_log_msg(E_LOG_INFO,FRESULT_S[result]);

        continue;

    }

    ax_log_msg(E_LOG_INFO,"\n\rf_open Sucessfull");

    result=f_lseek(&file_ptr,FILE_SIZE_LIMIT_1GB);

    if(result!=FR_OK){

        ax_log_msg(E_LOG_INFO,"\n\rf_lseek Failed for preallocation\n\rResult code");
        ax_log_msg(E_LOG_INFO,FRESULT_S[result]);

        f_close(&file_ptr);

        continue;

    }

    ax_log_msg(E_LOG_INFO,"\n\rf_lseek Sucessfull for preallocation");

    f_lseek(&file_ptr,0);

    bytes_to_write=sizeof(messages[file_count]);

    write_count=0;

    while( (f_tell(&file_ptr) < FILE_SIZE_LIMIT_1GB )){

        result=f_write(&file_ptr,messages[file_count],bytes_to_write,&bytes_written);

        if(result==FR_OK){
            ++write_count;

            if(write_count%50==0){

                f_sync(&file_ptr);
            }

        }else{

            ax_log_msg(E_LOG_INFO,"\n\rWrite failed\n\rFRESULT=");
            ax_log_msg(E_LOG_INFO,FRESULT_S[result]);

            break;

        }

    }

    f_close(&file_ptr);


}

Note :

1. ax_log_msg() is part of the device firmware to print on console.
2. FRESULT_S[result] is used to convert the enum result code to a string.

If there is any data missing, please do mention it.

Thank You

Answer 1

You probably need to buffer an entire block of data, perhaps 4 KB, to avoid flashing an entire block with every flush. But, the filesystem or driver should do this for you, as long as you don't call fflush explicitly, which is the real lesson.

Why do you need it to be synced so often? Perhaps a timer would work better than an interval per number of records?