mpi send/recv multiple variable length arrays in a struct

Question

I need to be able to send and receive two uint64_t arrays to all ranks from rank 0.

I have a typedef struct as follows:

typedef struct _mpi_data_t {
    uint64_t *array1;
    uint64_t *array2;
    size_t array1_size;
    size_t array2_size;
} mpi_data_t;

In rank 0, I populate the arrays and calculate the displacements to create the mpi data type.

MPI_Get_address(&mpi_data_send, &address[0]);
MPI_Get_address(&mpi_data_send.array1[0], &address[1]);
MPI_Get_address(&mpi_data_send.array2[0], &address[2]);
MPI_Get_address(&mpi_data_send.array1_size, &address[3]);
displacements[0] = address[1] - address[0];
displacements[1] = address[2] - address[0];
displacements[2] = address[3] - address[0]; 

I also set the block count array as follows:

int block_count[3] = {mpi_data_send.array1_size, mpi_data_send.array2_size, 2};

My problem is that when receiving this data type, other ranks can't have created the same data type spec with the same displacements and block counts because the arrays were generated by rank 0, the sender.

How can I send and receive these two dynamically sized arrays?

Answer 1

You can use MPI_Probe and MPI_Get_count to find receiving message length.
You don't provide a format of send and receive in your code but as a suggestion, you can send array separately and receive them. This C example can help:

// Author: Wes Kendall
// Copyright 2011 www.mpitutorial.com
// This code is provided freely with the tutorials on mpitutorial.com. Feel
// free to modify it for your own use. Any distribution of the code must
// either provide a link to www.mpitutorial.com or keep this header intact.
//
// Example of using MPI_Probe to dynamically allocated received messages
//
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(int argc, char** argv) {
  MPI_Init(NULL, NULL);

  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  if (world_size != 2) {
    fprintf(stderr, "Must use two processes for this example\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
  }
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  int number_amount;
  if (world_rank == 0) {
    const int MAX_NUMBERS = 100;
    int numbers[MAX_NUMBERS];
    // Pick a random amont of integers to send to process one
    srand(time(NULL)+world_rank);
    number_amount = (rand() / (float)RAND_MAX) * MAX_NUMBERS;
    // Send the amount of integers to process one
    MPI_Send(numbers, number_amount, MPI_INT, 1, 0, MPI_COMM_WORLD);
    printf("0 sent %d numbers to 1\n", number_amount);
  } else if (world_rank == 1) {
    MPI_Status status;
    // Probe for an incoming message from process zero
    MPI_Probe(0, 0, MPI_COMM_WORLD, &status);
    // When probe returns, the status object has the size and other
    // attributes of the incoming message. Get the size of the message.
    MPI_Get_count(&status, MPI_INT, &number_amount);
    // Allocate a buffer just big enough to hold the incoming numbers
    int* number_buf = (int*)malloc(sizeof(int) * number_amount);
    // Now receive the message with the allocated buffer
    MPI_Recv(number_buf, number_amount, MPI_INT, 0, 0, MPI_COMM_WORLD,
         MPI_STATUS_IGNORE);
    printf("1 dynamically received %d numbers from 0.\n",
       number_amount);
    free(number_buf);
  }
  MPI_Finalize();
}

in this example, random length array (less than 100) allocated in rank 0 and sent to rank 1.