MPI code hangs at finalize with more than 2 nodes/processes

Question

The following code is suppose to go send a series of messages to each node, and report the time it takes for each communication. At the moment it exits fine with processes, but if I run with more than 2 processes it hangs on the last exchange.

I've put statements in previous versions to check where it hangs, I am 90% sure that it is the MPI_FINALIZE statement, but I can't quite figure out why. Any ideas?

 #include <stdio.h>
 #include "/usr/include/mpich2/mpi.h"
 #define ping 101
 #define pong 101
 float buffer[100000];
int main (int argc, char *argv[]){
    int error, rank, size; //mpi holders
    int i, j, k; //loops
    extern float buffer[100000]; //message buffer
    int length; //loop again
    double start, final, time;
    extern float buffer[100000];


    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
      if(rank!=0){
            MPI_Status status;
          for(i=1;i<size;i++){
           for(length=1;length<=30000;length+=1000){
            for(j=0;j<100;j++){
             MPI_Recv(buffer, length, MPI_FLOAT, 0, ping, MPI_COMM_WORLD, &status);
             MPI_Send(buffer, length, MPI_FLOAT, 0, pong, MPI_COMM_WORLD);
             }
            }
          }
         }
      if(rank==0){
         MPI_Status status;
         for(i=1;i<size;i++){
           for(length=1;length<=30000;length+=1000){
            start = MPI_Wtime();
            for(j=0;j<100;j++){
             MPI_Send(buffer, length, MPI_FLOAT, i, ping, MPI_COMM_WORLD);
             MPI_Recv(buffer, length, MPI_FLOAT, MPI_ANY_SOURCE, pong, MPI_COMM_WORLD, &status);
             }
            final = MPI_Wtime();
            time = final-start;
            printf("%s\t%d\t%f\n", "Node", i, time);
            }
         }
     }

  MPI_Finalize();
  return 0;
  }

Answer 1

You have an extra loop in non-zero ranks:

  if(rank!=0){
        MPI_Status status;
      for(i=1;i<size;i++){   <-----------
       for(length=1;length<=30000;length+=1000){
        for(j=0;j<100;j++){
         MPI_Recv(buffer, length, MPI_FLOAT, 0, ping, MPI_COMM_WORLD, &status);
         MPI_Send(buffer, length, MPI_FLOAT, 0, pong, MPI_COMM_WORLD);
         }
        }
      }                      <-----------
     }

With 2 ranks that loop executes a single iteration but with more than two ranks it executes size-1 iterations. Since rank 0 sends the messages only once per rank, you have to remove that loop.