OpenMPI MPI_Send vs Intel MPI MPI_Send

Question

I have a code which I compile and run using openmpi. Lately, I wanted to run this same code using Intel MPI. But my code is not working as expected. I digged into the code and found out that MPI_Send behaves differently in both implementation.

I got the advice from the different forum to use MPI_Isend instead of MPi_Send from different forum. But that requires hell lot of work to modify the code. Is there any workaround in Intel MPI to make it work just like in OpenMPI. May be some Flags or Increasing Buffer or something else. Thanks in advance for your answers.

int main(int argc, char **argv) {
    int numRanks;
    int rank;
    char cmd[] = "Hello world";
    MPI_Status status;

    MPI_Init (&argc, &argv);
    MPI_Comm_size (MPI_COMM_WORLD, &numRanks);
    MPI_Comm_rank (MPI_COMM_WORLD, &rank);
    if(rank == 0) {
            for (int i=0; i< numRanks; i++) {
                    printf("Calling MPI_Send() from rank %d to %d\n", rank, i);
                  MPI_Send(&cmd,sizeof(cmd),MPI_CHAR,i,MPI_TAG,MPI_COMM_WORLD);
                    printf("Returned from MPI_Send()\n");
            }
    }
    MPI_Recv(&cmd,sizeof(cmd),MPI_CHAR,0,MPI_TAG,MPI_COMM_WORLD,&status);
    printf("%d receieved from 0 %s\n", rank, cmd);

    MPI_Finalize();
}

OpenMPI Result

# mpirun --allow-run-as-root  -n 2  helloworld_openmpi
Calling MPI_Send() from rank 0 to 0
Returned from MPI_Send()
Calling MPI_Send() from rank 0 to 1
Returned from MPI_Send()
0 receieved from 0 Hello world
1 receieved from 0 Hello world

Intel MPI Result

# mpiexec.hydra -n 2 /root/helloworld_intel

Calling MPI_Send() from rank 0 to 0

Stuck at MPI_Send.

Answer 1

It is incorrect to assume MPI_Send() will return before a matching receive is posted, so your code is incorrect with respect to the MPI Standard, and you are lucky it did not hang with Open MPI.

MPI implementation usually eager-send small messages so MPI_Send() can return immediately, but this is an implementation choice not mandated by the standard, and "small" message depends on the library version, the interconnect you are using and other factors.

The only safe and portable choice here is to write correct code.

FWIW, MPI_Bcast(cmd, ...) is a better fit here, assuming all ranks already know the string length plus the NUL terminator.

Last but not least, the buffer argument is cmd and not &cmd.