Search code examples
mpicommunicationhpc

Collective communications along directions in MPI cartesian topology

I have a 3D cartesian topology of nx by ny by nz processes. There are mathematical calculations that involve at the same time only "pencils" of processors. In the case of a 3 by 3 by 3 matrix of processes, ranked from 0 to 26, process 4 is involved in three operations:

  1. with processes 13 and 22 along the first direction
  2. with processes 1 and 7 along the second direction
  3. with processes 3 and 5 along the third direction

That mathematical operations require both point to point and collective communications between processes belonging to the same pencil.

For what concerns point to point communications, I used MPI_CART_SHIFT to make each process know the ranks of neighboring processes. (Then I'm going to use MPI_SENDRECV.)

For what concerns collective communications, how to perform such communications? I think a solution could be to define "pencil" communicators, which would be in number of nx*ny + nx*nz + ny*nz (this number of communicators required is asymptotically small with respect to the number of processes, as the number of processes per direction grows).

Would this be the only way? Is there no standard subroutine relying on the cartesian communicator to perform such collective communications?

Solution

  • The neighborhood collectives are really the only routines that can directly exploit the connectivity information for cartesian topologies. However, they would treat all the directions (x, y, z) in the same way so wouldn't help you with your pencil scheme.

    I think the only way to do this is as you suggest, i.e. construct a complete set of pencil communicators. Note that MPI does give you an easy way to do this by calling MPI_Cart_sub on the cartesian communicator. Once you've constructed the pencil communicators then you would also have the option of using neighborhood collectives on the pencils rather than point-to-point, but for a 1D communicator it's not clear that this has many advantages to computing neighbours by hand as you do at present.