How to write a multi-dimensional array of structs to disk using MPI I/O?

Question

I am trying to write an array of complex numbers to disk using MPI I/O. In particular, I am trying to achieve this using the function MPI_File_set_view so that I can generalise my code for higher dimensions. Please see my attempt below.

struct complex
{
    float real=0;
    float imag=0;
};

int main(int argc, char* argv[])
{
    /* Initialise MPI parallel branch */
    int id, nprocs;
    MPI_Init(&argc, &argv);
    MPI_Comm comm = MPI_COMM_WORLD;
    MPI_Comm_rank(comm, &id);
    MPI_Comm_size(comm, &nprocs);

    /* Create a datatype to represent structs of complex numbers */
    MPI_Datatype MPI_Complex;
    const int    lengths[2] = { 1, 1 };
    MPI_Datatype types[2] = { MPI_FLOAT, MPI_FLOAT };
    MPI_Aint     displacements[2], base_address;
    complex      dummy_complex;
    MPI_Get_address(&dummy_complex, &base_address);
    MPI_Get_address(&dummy_complex.real, &displacements[0]);
    MPI_Get_address(&dummy_complex.imag, &displacements[1]);
    displacements[0] = MPI_Aint_diff(displacements[0], base_address);
    displacements[1] = MPI_Aint_diff(displacements[1], base_address);
    MPI_Type_create_struct(2, lengths, displacements, types, &MPI_Complex);
    MPI_Type_commit(&MPI_Complex);

    /* Create a datatype to represent local arrays as subarrays of a global array */
    MPI_Datatype MPI_Complex_array;
    const int global_size[1] = { 100 };
    const int local_size[1] = { (id < nprocs-1 ? 100/nprocs : 100/nprocs + 100%nprocs) };
    const int glo_coord_start[1] = { id * (100/nprocs) };
    MPI_Type_create_subarray(1, global_size, local_size, glo_coord_start, MPI_ORDER_C,
                             MPI_Complex, &MPI_Complex_array);
    MPI_Type_commit(&MPI_Complex_array);

    /* Define and populate an array of complex numbers */
    complex *z = (complex*) malloc( sizeof(complex) * local_size[0] );
    /* ...other stuff here... */

    /* Write local data out to disk concurrently */
    MPI_Offset offset = 0;
    MPI_File file;
    MPI_File_open(comm, "complex_nums.dat", MPI_MODE_CREATE|MPI_MODE_WRONLY, MPI_INFO_NULL, &file);
    MPI_File_set_view(file, offset, MPI_Complex, MPI_Complex_array, "external", MPI_INFO_NULL);
    MPI_File_write_all(file, z, local_size[0], MPI_Complex, MPI_STATUS_IGNORE);
    MPI_File_close(&file);

    /* ...more stuff here... */
}

However, with the above code, only the local data from the process labelled id = 0 is being saved to disk. What am I doing wrong and how can I fix this? Thank you.

N.B. Please note that for this one dimensional example, I can avoid the problem by giving up on MPI_File_set_view and using something simpler like MPI_File_write_at_all. Nevertheless, I have not solved the underlying problem because I still don't understand why the above code does not work, and I would like a solution that can be generalised for multi-dimensional arrays. Your help is much appreciated in this regard. Thank you.

Answer 1

The default behavior is not to abort when a MPI-IO subroutine fails. So unless you change this default behavior, you should always test the error code returned by MPI-IO subroutines.

In your case, MPI_File_set_view() fails because external is not a valid data representation. I guess this is a typo and you meant external32.