Search code examples
carraysfilefwritefread

Storing a huge array in a file, row by row, results in a damaged file

I have an input array A stored in the memory, which is used to produce another array, much bigger B. However, since B is a huge array, I don't really want to store it in the memory, but to save it locally into a file (using fwrite). For this, I calculate every iteration the ith row and append it to the output file. That way, I only need to store one row at a time in the memory, and eventually, an output file is created, with all the data I needed.

The output file seems to be in proper size, considering the number of items it consists of. Nevertheless, when I try to read back fragments from the output file using fread, (for instance, retrieve the first 2000 items), only the first 23 items are retrieved.

This is the primary function to create the output file:

void exportCovMatrix(char *outputString, double *inputStdMatrix, int colDim, int rowDim) {
    double *covRow = calloc(rowDim, sizeof(double));
    int i, j, n;
    FILE *output;
    fclose(fopen(outputString, "w"));
    output = fopen(outputString, "a");
    assert(covRow != NULL);
    assert(output != NULL);
    for (i = 0; i < rowDim; i++) {
        for (j = 0; j < rowDim; j++)
            covRow[j] = dotProduct(&inputStdMatrix[i * colDim], &inputStdMatrix[j * colDim], colDim);
        n = fwrite(covRow, sizeof(double), rowDim, output);
        assert(n == rowDim);
    }
    fclose(output);
    free(covRow);
}

This is another function, that reads the given output file:

double *calculateNextB(char* inputString, double* row, int dim){
    FILE* input = fopen(inputString, "r");
    int i, j;
    assert(input != NULL);
    for(i = 0; i <= dim; i++){
        j = fread(row, sizeof(double), dim, input);
        printf("%d items were read.\n", j);
    }
    ...
}

I'd appreciate any help in solving this issue. Thanks!

Solution

  • You open the file respectively with

    fclose(fopen(outputString, "w"));

    and

    FILE* input = fopen(inputString, "r");

    But as explained for example here

    In order to open a file as a binary file a "b" character has to be included in the mode string.

    (I know it is a C++ source, but in some system it is true, though it's not in many POSIX systems, as explained in https://linux.die.net/man/3/fopen )