Search code examples
ctimeparallel-processingmpidistributed-computing

Different time measurements with MPI

I am getting different time measurements every time I run my MPI application. I am measuring time like this:

MPIt1 = MPI_Wtime();

// do work

MPIt2 = MPI_Wtime();
MPIelapsed = MPIt2 - MPIt1;

The behavior is same, with or without MPI_Barrier(MPI_COMM_WORLD);. The result of the program, of course, is correct every time the executable runs.

Check the extreme difference I got for a 5x5 matrix:

0.00025 seconds // first run
0.10728 seconds // second run

Moreover, the problem is occurring with larger matrices, like this 2000x2000:

0.55996 seconds // first run
0.83657 seconds // second run

Am I measuring the time in a wrong way? Or is this a knowing issue with MPI? Note that the time is reported only from the master node.

I am running the program in another computer, where I have logged in with ssh. I recall getting faster execution when I was actually sitting in that computer. I am the only user logged in:

gsamaras@pythagoras:~/konstantis/cholesky$ who
gsamaras my internet connection
gsamaras@pythagoras:~/konstantis/cholesky$

EDIT with many runs:

1.6035
0.422967
1.53891
0.347107
0.761991
1.22824
0.934829
0.45618
0.794225
0.554099
1.09323
0.421858
0.621015
0.889867
0.613867
0.459144
2.02273
0.67689
2.2390
1.0448
0.751333

EDIT_2

  1. I got the timings, by calling the program many times in a run.sh.
  2. I am running the program in one machine, with 4 cores.
  3. I am using MPICH2.
Solution

  • From a colleague and Patrick I concluded that:

    1. The startup time for MPI can vary a lot.
    2. Relatively small execution times are not trust-worthy and may vary a lot.