A not safe MPI non-blocking communication example?

Question

I am implementing MPI non-blocking communication inside my program. I see on MPI_Isend man_page, it says:

A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not modify any part of the send buffer after a nonblocking send operation is called, until the send completes.

My code works like this:

// send messages
if(s > 0){

    MPI_Requests s_requests[s];
    MPI_Status   s_status[s];

    for(int i = 0; i < s; ++i){

        // some code to form the message to send
        std::vector<doubel> send_info;

        // non-blocking send
        MPI_Isend(&send_info[0], ..., s_requests[i]);
    }

    MPI_Waitall(s, s_requests, s_status);
}

// recv info
if(n > 0){    // s and n will match

    for(int i = 0; i < n; ++i){

        MPI_Status status;

        // allocate the space to recv info
        std::vector<double> recv_info;

        MPI_Recv(&recv_info[0], ..., status)
    }

}

My question is: am I modify the send buffers since they are in the inner curly brackets (the send_info vector get killed after the loop finishes)? Therefore, this is not a safe communication mode? Although my program works fine now, I still being suspected. Thank you for your reply.

Answer 1

There are two points I want to emphasize in this example.

The first one is the problem I questioned: send buffer gets modified before MPI_Waitall. The reason is what Gilles said. And the solution could be allocated a big buffer before the for loop, and use MPI_Waitall after the loop is finished or put MPI_Wait inside the loop. But the latter one is equivalent to use MPI_Send in the sense of performance.

However, I found if you simply transfer to blocking send and receive, a communication scheme like this could cause deadlock. It is similar to the classic deadlock:

if (rank == 0) {
      MPI_Send(..., 1, tag, MPI_COMM_WORLD);
      MPI_Recv(..., 1, tag, MPI_COMM_WORLD, &status);
 } else if (rank == 1) {
      MPI_Send(..., 0, tag, MPI_COMM_WORLD);
      MPI_Recv(..., 0, tag, MPI_COMM_WORLD, &status);
 }

And the explaination could be found here.

My program could cause a similar situation: all the processors called MPI_Send then it is a deadlock.

So my solution is to use a large buffer and stick to non-blocking communication scheme.

#include <vector>
#include <unordered_map>

// send messages
if(s > 0){

    MPI_Requests s_requests[s];
    MPI_Status   s_status[s];

    std::unordered_map<int, std::vector<double>> send_info;

    for(int i = 0; i < s; ++i){


        // some code to form the message to send
        send_info[i] = std::vector<double> ();

        // non-blocking send
        MPI_Isend(&send_info[i][0], ..., s_requests[i]);
    }

    MPI_Waitall(s, s_requests, s_status);
}

// recv info
if(n > 0){    // s and n will match

    for(int i = 0; i < n; ++i){

        MPI_Status status;

        // allocate the space to recv info
        std::vector<double> recv_info;

        MPI_Recv(&recv_info[0], ..., status)
    }

}