Search code examples
mpihpcmpich

Tau2Slog2 not able to process 6gb tau.trc files

I am profiling my code using the TAU profiler. I am using tau_exec at runtime. It generates trace files. Some of which are in gigabytes. tau_treemerge.pl merges and generates a tau.trc which is 6GB. tau2slog2 now fails complaining about the heap space.

It would be helpful if anybody can show how to reduce the size of trace files.

Following is the way I am running the code:

mpirun -n  64 tau_exec ./a.out 
tau_treemerge.pl;
tau2slog2 tau.tr tau.edf -o tau.slog2
Solution

  • I was able to solve the issue by increasing the heap size of the JVM.

    java -Xmx50000m -Xms32000m -cp /tau/x86_64/lib/TAU_tf.jar:/tau/x86_64/lib/traceTOslog2.jar:/tau/x86_64/lib/tau2slog2.jar edu/uoregon/tau/Tau2Slog2  tau.trc tau.edf -o tau.slog2

    Obviously it is a workaround and not an elegant solution. So to reduce the tau.trc filesize I have added more filtering parameters during the instrumentation.

    Also I first just profiled the code export TAU_PROFILE=1 and then ran pprof and figured out which MPI function is called enormously then throttled those functions to further reduce the file size.