Interleaving binary data from different processors on MPI-IO

Question

I'm trying to write a binary file using MPI I/O (MPI-2.0, mpich2).

Below is a minimal example where 2 files 'chars' and 'ints' should be printed as '0123...' and 'abcd...' respectively.

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) 
{
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    // MPI I/O
    MPI_File file;
    MPI_Status status;
    int offset = world_rank;

    // MPI I/O with integers
    MPI_File_open (
        MPI_COMM_WORLD,     // MPI_Comm comm
        "ints",             // char *filename
    MPI_MODE_CREATE |       // int amode
    MPI_MODE_RDWR,
    MPI_INFO_NULL,          // MPI_Info info
    &file                   // MPI_File *fh
    );
    int buf1 = world_rank;
    MPI_File_write_at (
        file,               // MPI_File fh
        offset,             // MPI_Offset offset
        &buf1,              // void *buf
        1,                  // int count
        MPI_INT,            // MPI_Datatype datatype
        &status             // MPI_Status *status
    );
    MPI_File_close (&file);

    // MPI I/O with chars
    MPI_File_open (
        MPI_COMM_WORLD,     // MPI_Comm comm
        "chars",            // char *filename
    MPI_MODE_CREATE |       // int amode
    MPI_MODE_RDWR,
    MPI_INFO_NULL,          // MPI_Info info
    &file                   // MPI_File *fh
    );
    char buf2 = 'a' + (char)world_rank;
    MPI_File_write_at (
        file,               // MPI_File fh
        offset,             // MPI_Offset offset
        &buf2,              // void *buf
        1,                  // int count
        MPI_CHAR,           // MPI_Datatype datatype
        &status             // MPI_Status *status
    );
    MPI_File_close (&file);


    // Finalize the MPI environment.
    MPI_Finalize();
    return 0;
}

I get the right result with chars,

> od -c chars
0000000   a   b   c   d
0000004

but with integers it works only if np = 0. For np > 0, I get results which I don't understand:

np = 2:

> od -i ints
0000000         256           0
0000005

np = 3:

> od -i ints
0000000      131328           0
0000006

etc.

Is my code wrong, or is 'od -i' simply inappropriate for displaying a binary file with integers?

Thanks,

Juri

Answer 1

Unless you've set a file view with an elementary type, the offset index is actually in units of bytes - that is, the default etype is single bytes. This is actually very sensible, but it would be helpful if it were documented in the manpages.

So changing this line:

MPI_File_write_at (
    file,               // MPI_File fh
    offset,             // MPI_Offset offset

to this:

MPI_File_write_at (
    file,               // MPI_File fh
    offset*sizeof(int), // MPI_Offset offset

Gives you the answer that you want:

$ od -i ints 
0000000           0           1           2
0000014

On the other hand, if you'll only be using integers in this file, easier is to set a file view:

MPI_File_set_view( file, 0, MPI_INT, MPI_INT, "native", MPI_INFO_NULL);
MPI_File_write_at (
    file,               // MPI_File fh
    offset,             // MPI_Offset offset

Also works. And in general, you'll want to use different file views rather than explicit offsets for each processor.