Search code examples
c++io

fstream faster than fopen? and gets faster as buffers increase?


I've been trying to write file data as fast as possible.

  • I've increased the buffer size to reduce i/o operations.
  • I've tested with both fstream and fopen.

For some reason fstream is faster than fopen.

  • on a 64 byte buffer it's ~1.3 times faster
  • on a 8192 byte buffer it's ~4.8 times faster.

I've been hearing that C's file I/O is faster (which makes sense)
<fstream> includes <stdio.h> yet i can't get fopen to perform as fast.

NOTE (old questions):

  • my fopen was 2 times slower than fstream because i used fprintf (thanks jamesdlin)
  • fstream buffer wasn't changing since you have to set it before opening (thanks Paul Sanders)

also realized fstream.put(char) is faster than fstream << char
(otherwise fopen is faster than fstream if the buffer is < ~256)

Here's my testing:

#include <iostream>
#include <fstream>
#include <ctime>

int filesize; // total bytes (individually "put" in buffered stream)
int buffsize; // buffer size

void writeCPP(){
    std::ofstream file;
    char buffer[buffsize]; file.rdbuf()->pubsetbuf(buffer,buffsize);    // set buffer (before opening)
    file.open("test.txt",std::ios::binary);                             // open file
    for(int i=0; i<filesize; i++) file.put('a');                        // write bytes
    file.close();                                                       // close
}

void writeC(){
    FILE* file=fopen("test.txt","wb");                                  // open file
    char buffer[buffsize]; setvbuf(file,buffer,_IOFBF,buffsize);        // set buffer
    for(int i=0; i<filesize; i++) fputc('a',file);                      // write bytes
    fclose(file);                                                       // close
}

#define getTime() double(clock())/CLOCKS_PER_SEC // good enough

double start;

void test(int s){ // C++ vs C (same filesize / buffsize)
    buffsize=s;
    std::cout<<"  buffer: "<<buffsize<<"\t"<<std::flush;

    start=getTime();
    writeCPP();
    std::cout<<"  C++: "<<getTime()-start<<",\t"<<std::flush;

    start=getTime();
    writeC();
    std::cout<<" C: "<<getTime()-start<<std::endl;
}

#define MB (1024*1024)

int main(){
    filesize=10*MB;
    std::cout<<"size: 10 MB"<<std::endl;

    // C++ fstream faster
    test(64);   // C++ 0.86 < C 1.11 (1.29x faster)
    test(128);  // C++ 0.44 < C 0.79 (1.80x faster) (+0.51x)
    test(256);  // C++ 0.27 < C 0.63 (2.33x faster) (+0.53x)
    test(512);  // C++ 0.19 < C 0.56 (2.94x faster) (+0.61x)
    test(1024); // C++ 0.15 < C 0.52 (3.46x faster) (+0.52x)
    test(2048); // C++ 0.14 < C 0.51 (3.64x faster) (+0.18x)
    test(4096); // C++ 0.12 < C 0.49 (4.08x faster) (+0.44x)
    test(8192); // C++ 0.10 < C 0.48 (4.80x faster) (+0.72x)
}

Solution

  • In WriteCPP, you have to set the buffer before opening the file, like so:

    std::ofstream file;
    char buffer[BUFF]; file.rdbuf()->pubsetbuf(buffer, BUFF);   // set buffer
    file.open ("test.txt", std::ios::binary);                   // open file
    

    Then you get the sort of results that you might you expect (times are for writing 20MB with the buffer sizes shown):

    writeCPP, 32: 2.15278
    writeCPP, 128: 1.21372
    writeCPP, 512: 0.857389
    

    I also benchmarked WriteC with your change from fprintf to fputc and got the following (again writing 20MB):

    writeC, 32: 1.41433
    writeC, 128: 0.524264
    writeC, 512: 0.355097
    

    Test program is here:

    https://wandbox.org/permlink/F2H2jcrMVsc5VNFf