MPI functions appear to execute out of order compared to printf()

Question

I am having a very strange code using MPI, in which statements appear to be executed in the wrong order. Specifically, the MPI statement appears to be executing before the printf even though it comes after it in the code.

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv)
{
    int numProcs, rank, data;
    MPI_Status status;
    
    // Initialize the MPI library
    MPI_Init(&argc, &argv);

    // Get entity identification
    MPI_Comm_size(MPI_COMM_WORLD, &numProcs);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    // Do something different in each a rank
    if (rank == 0) {
        // Get the data from rank   1
    // with tag                    0
    printf("rank = %d\tGet the data from rank 1 with tag 0\n", rank);
    MPI_Recv(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &status);
    } else if (rank == 1) {
        // Send the data to rank    0
    // with tag                    0
    printf("rank = %d\tSend the data to rank 0 with tag 0\n", rank);
        MPI_Send(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
    }

    printf("rank %d finishing\n", rank);

    // Clean up the MPI library
    MPI_Finalize();

    return 0;
}

This is the output that is being generated:

$ mpirun -n 2 ./a.out
rank = 0    Get the data from rank 1 with tag 0
rank 0 finishing
rank = 1    Send the data to rank 0 with tag 0
rank 1 finishing

it seems that rank 0 does the printf, then it gets the data from rank 1, and then it finishes … and then rank 1 does the printf? But since rank 1 has to do the printf before it actually sends the data to rank 0, how can it be that the rank 0 already got the data and finished?

Answer 1

Screen output gets buffered by the OS, and then has to make its way through ssh tunnels to the process that started the MPI run. As a result, screen output can arrive in all sorts of orders. There is basically no way to get neatly ordered screen output, other than sending all text to process zero and printing from there in the correct order.