I have the following simple MPI code:
#include <iostream>
#include <mpi.h>
int main() {
struct test {
int rank = 1;
double begin = 0.5;
double end = 0.5;
};
MPI_Init(NULL, NULL);
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_File fh;
MPI_Offset offset;
MPI_Status status;
MPI_Datatype TIMEDATA_DATATYPE;
// Create datatype
int blocklen[3] = {1,1,1};
MPI_Datatype type[3] = { MPI_INT, MPI_DOUBLE, MPI_DOUBLE };
MPI_Aint disp[3] = {0, 4, 12};
MPI_Type_create_struct(3, blocklen, disp, type, &TIMEDATA_DATATYPE);
MPI_Type_commit(&TIMEDATA_DATATYPE);
// Open file
offset = 20*world_rank;
MPI_File_open(MPI_COMM_WORLD, "test.bin", MPI_MODE_CREATE | MPI_MODE_WRONLY,
MPI_INFO_NULL, &fh);
// Write to file
test t1, t2;
MPI_File_write_at(fh, offset, &t1, 1, TIMEDATA_DATATYPE, &status);
MPI_File_write_at(fh, offset, &t2, 1, TIMEDATA_DATATYPE, &status);
// Close file
MPI_File_close(&fh);
MPI_Finalize();
return 0;
}
Run and compiled with
mpic++ Test.cpp
mpirun -n 2 a.out
So in the code above, I basically want to write to the same file twice. Each time from a different process. The data is a struct of the format int, double, double so 4+8+8=20 bytes of data. We have two objects of that struct but both are initialized with the same values. So we write two blocks of 20 bytes of data into the file test.bin and I expect to see this "symmetry" in the binary representation but I get: (I provide two different outputs of two different cli tools.)
$ xxd test.bin
00000000: 0100 0000 fe7f 0000 0000 0000 0000 e03f ...............?
00000010: 0000 e03f 0100 0000 ff7f 0000 0000 0000 ................
00000020: 0000 e03f 0000 0000 ...?....
$ hexdump test.bin
0000000 0001 0000 7ffe 0000 0000 0000 0000 3fe0
0000010 0000 0000 0001 0000 7fff 0000 0000 0000
0000020 0000 3fe0 0000 0000
0000028
Now, if we look at the output of xxd we see:
First 20 bytes:
Second 20 bytes:
Now basically, I'm not fully sure why the doubles here differ.
Instead of:
MPI_Aint disp[3] = {0, 4, 12};
do the following:
disp[0] = offsetof(test, rank);
disp[1] = offsetof(test, begin);
disp[2] = offsetof(test, end);
do not forget to include "#include <stddef.h>", and to adapt the offset (i.e., offset = sizeof(test) * world_rank;)
Do not try to manually hardcode the offsets, it is better to use offsetof from <stddef.h> to figure it out for you. There is no guarantee that the padding used inside structures will match the values that you have hardcoded in your array for the displacements (i.e., disp)
Structure padding is a concept in C that adds the one or more empty bytes between the memory addresses to align the data in memory. (source). To a better understanding of padding look into this SO thread