Search code examples
mpinonblocking

non-blocking MPI, periodic boundaries with 2 processors

I'm writing a halo-exchange routine that looks like this:

halo exchange diagram, example with 3 processors

where non-blocking send/recv calls are used as described by the following pseudo-code:

  1. mpi_irecv from left_neighbour ( + same for right neighbour)
  2. pack data and mpi_isend to left_neighbour ( + same for right neighbour)
  3. mpi_wait(recv_request from left_neighbour) and unpack ( + same for right neighbour)
  4. mpi_wait(send_request to left_neighbour) ( + same for right neighbour)

This works fine in all cases, except for the specific case where I request two processors, and the boundaries are periodic. In this situation the data is mixed, i.e. recv_buffer_left has the data I expect recv_buffer_right to have, and v.v.

The rank for the left & right neighbours are the same in this case, so I assume I'm missing some subtlety in the request handles/tags? The code is below, would appreciate any thoughts!

! --- 1. issue non-blocking receives

if ( left_neighbour_exists ) call mpi_irecv( recv_buffer_left, buffer_size, MPI_REAL, left_neighbour, itag, world_comm, request_recv_left, ierr )
if ( right_neighbour_exists ) call mpi_irecv( recv_buffer_right, buffer_size, MPI_REAL, right_neighbour, itag, world_comm, request_recv_right, ierr )

! --- 2. pack for sending, then issue non-blocking sends

call pack_data( my_data, send_buffer_left, send_buffer_right )
!
if ( left_neighbour_exists ) call mpi_isend( send_buffer_left, buffer_size, MPI_REAL, left_neighbour, itag, world_comm, request_send_left, ierr )
if ( right_neighbour_exists ) call mpi_isend( send_buffer_right, buffer_size, MPI_REAL, right_neighbour, itag, world_comm, request_send_right, ierr )

! --- 3. wait for receives, then unpack

if ( left_neighbour_exists ) then
    !
    call mpi_wait( request_recv_left, status, ierr )
    !
    call unpack_data( recv_buffer_left, my_data )
    !
endif
!
if ( right_neighbour_exists ) then
    !
    call mpi_wait( request_recv_right, status, ierr )
    !
    call unpack_data( recv_buffer_right, my_data )
    !
endif

! --- 4. wait for sends

if ( left_neighbour_exists ) call mpi_wait( request_send_left, status, ierr )
if ( right_neighbour_exists ) call mpi_wait( request_send_right, status, ierr )
Solution

  • You need to be careful about message ordering. For more than 2 processes, there is no ambiguity as your up and down neighbours are the same. On two processes, however, your up and down neighbours are the same process so you are sending two messages to, and receiving two messages from, the same process.

    The trick is to switch your sends around so you send right first then send left. The first send will match the first receive, which is a receive from the left.