Search code examples
mpitracehpc

Collecting an MPI Trace

How can I collect an MPI communication trace on Supercomputers?

I need text files with details of each message (say sender, receiver, size, etc.) that I can parse. I was using following command for Intel MPI and do not see any text files.

mpirun -trace -n 4 -trace-pt2pt -trace-collectives ./myApp
Solution

  • I am not familiar with Intel MPI's integrated solution.

    There is a number of tools that provide MPI tracing.

    Performance focussed:

    Correctness checking:

    I recommend to not roll your own solution, because it's not straight forward to match receives to sends and you might run into timing issues because timers are not synchronized across nodes.

    You could e.g. trace a run using Score-P, and then use the otf2-print command on the trace to get the text output you wanted. Or you can use the OTF2 reader library and develop a tool on top of it. Here is a short tutorial on how to run Score-P, starting at slide 17