Fortran MPI_Isend and MPI_Irecv

Question

I have a problem with MPI_Isend and MPI_Irecv: The receiving vectors are never sent correctly. The code is written in Fortran.

Every process has number of touching processes to which I want to send some values. The values that I want to send consist of 4 vectors that are part of a type called variables for each process.

Here is the code that I use:

program isend_test
use mpi

real, dimension(:,:,:), allocatable :: receivedValues
real, dimension(:,:), allocatable :: sendReals
integer, dimension(:,:), allocatable :: requestSend
integer, dimension(:,:), allocatable :: requestReceive
integer, dimension(:), allocatable :: neighbours
integer, dimension(mpi_status_size) :: status
integer :: ierr, currentNeighbour, k, me, nTasks, nValues, nNeighbours, addedNeighbours

call MPI_init(ierr)

call MPI_COMM_RANK(MPI_COMM_WORLD, me, ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, nTasks, ierr)

nNeighbours = 2

! Only 3 values for each variable to keep it simple
nValues = 3

allocate(receivedValues(nNeighbours,4,nValues))
allocate(sendReals(4,nValues))
allocate(requestSend(4,nNeighbours))
allocate(requestReceive(4,nNeighbours))
allocate(neighbours(2))

receivedValues = -9999

! Initializing neighbours - Every process is adjacent to every other process in this example
addedNeighbours = 0
do j = 0,2
    if (j == me) then
        cycle
    endif
    addedNeighbours = addedNeighbours + 1
    neighbours(addedNeighbours) = j
enddo

! fill in some values to send
do j = 1,4
   do i=1,nValues
      sendReals(j,i) = j + 10*me + 100*i
   enddo
enddo

do j = 1,4
    do i = 1,nNeighbours
        call mpi_isend(sendReals(j,:), nValues, mpi_real, neighbours(i), j, MPI_COMM_WORLD, requestSend(j,i), ierr)
        call mpi_irecv(receivedValues(i, j, :), nValues, mpi_real, neighbours(i), j, MPI_COMM_WORLD, requestReceive(j,i), ierr)
    enddo
enddo


do j = 1,4
    do i = 1,nNeighbours
        call mpi_wait(requestSend(j,i), status, ierr)
        call mpi_wait(requestreceive(j,i), status, ierr)
    enddo
enddo

write(*,*)receivedValues

call MPI_finalize(ierr)
end

I know that the datatypes are correct (they work with MPI_Sendand MPI_Recv) and the whole matching of the neighbours and the tags is correct as well because the code runs through correctly. However, if I set receivedValues = -9999in the beginning before synching, the values aren't changed.

I know the code could be done much more efficiently but I changed so much to find the error without success... Does anyone have an idea? It is probably a problem with the buffers, I just can't find it...

By the way: Sending and receiving sendReals(j,1) and neighbours(i), j, 1) does not work either...

Answer 1

The receive buffer is not contiguous in memory (since Fortran is column major)

mpi_irecv(receivedValues(i, j, :)

so this can only work if MPI_SUBARRAYS_SUPPORTED is .true. (and it is not in Open MPI, it might be the case with MPICH and mpi_f08 bindings).

Not only you are getting incorrect data, but you are also very likely causing silent data corruption.

You can either reorder the receivedValues array, or use a derived datatype and use receivedValues(i,j,1) as the receive buffer.