How to create a critical section only for some processes in Open/Microsoft MPI using C++?

Question

I am new to MPI and wanted to find out how to create a critical section for only a few of the running processes. I know that I can create one like this for ALL processes:

for (int i = 0; i < 1000; ++i)
{
    MPI_Barrier(MPI_COMM_WORLD);
    std::cout << "RANK " << rank << " PRINTS " << i << std::endl;
}

However this would not work if at least one process wouldn't trigger MPI_Barrier(). So let's say I did this:

if(rank > 0)
for (int i = 0; i < 1000; ++i)
{
    MPI_Barrier(MPI_COMM_WORLD);
    std::cout << "RANK " << rank << " PRINTS " << i << std::endl;
}

This would crash as the 0th processes would skip over the loop. So how do I allow all other processes to print synchronously and do something else with the 0th process (let's say wait for a message when all the printing is over)?

Answer 1

As @Gilles Gouaillardet pointed out, you can use MPI_Comm_split to create a communicator without rank 0. MPI_Barrier can be called with this new communicator to synchronise the remaining processes. Rank 0, after doing it's operations can call barrier over COMM_WORLD and wait for the remaining processes to call it.

Remaining processes, say 1 to N, can execute a code region sequentially (by the way not a MPI approach) by using a for loop iterating over 1 to N and an if block which executes the code region based on rank and MPI_Barrier as seen in the below example.

MPI_Comm_split(MPI_COMM_WORLD,color,key,newcomm) // create new comm without rank 0
if(rank>0) {
  for (int i = 1; i < size; ++i)
   {
    if(rank == i) { // Critical region. Only one rank can enter here at a time. Here it enters sequentially from 0 to size
       std::cout << "RANK " << rank << " PRINTS " << i << std::endl; // After calling Critical Session Part, call barrier to signal other processes.
       MPI_Barrier(newcomm);
    } else {
       MPI_Barrier(newcomm); // all other proesses (except rank == i) should wait and do not enter the critical region
    }
   }
       MPI_Barrier(MPI_COMM_WORLD);// This barrier will be called with rank 0         
} else {
       //do some work for rank 0
       // wait for the rest of the processes
       MPI_Barrier(MPI_COMM_WORLD);         
}

Another approach (without creating a new communicator) is you can use the concept of sending the message in ring (ring broadcast).

if (rank == 0) {
    value = 1; 
    MPI_Send( &value, 1, MPI_INT, rank + 1, 0, MPI_COMM_WORLD );
    // Do something useful
}
else {
    MPI_Recv( &value, 1, MPI_INT, rank - 1, 0, MPI_COMM_WORLD, 
          &status );
    if (rank < size - 1){ 
    //criticalregion
    MPI_Send( &value, 1, MPI_INT, rank + 1, 0, MPI_COMM_WORLD );
    }
}
MPI_Barrier(MPI_COMM_WORLD);         

Here, rank 0 send a message to rank 1 which in turn send to rank 2 and so on. This way, after receiving the message, the processes can execute the code sequentially (critical region) and then send the message to a higher rank and trigger the execution.