Search code examples
c++boostmpigmp

MPI and boost multiprecision/gmp

I tired to convert DBL type array to char array, broadcast it using MPI_Bcast and convert back to DBL array. The DBL array can be any of:

  1. double
  2. long double
  3. mpf_float_50 (it is the type defined in boost/multiprecision/ package).

The first two types work fine:

mpicxx -D_DBL main.cpp -o main && mpirun -np 2 main

or 

mpicxx -D_LDBL main.cpp -o main && mpirun -np 2 main

gives

before for rank=0
0
0.1
after for rank=1
0
0.1
after for rank=0
0
0.1

but for 

mpicxx -D_EDBL50 main.cpp -o main -lgmp && mpirun -np 2 main
one has
before for rank=0
0
0.1
after for rank=0
0
0.1
after for rank=1
0

[Fominskoe:06235] *** Process received signal ***
[Fominskoe:06235] Signal: Segmentation fault (11)
[Fominskoe:06235] Signal code: Address not mapped (1)
[Fominskoe:06235] Failing at address: 0x55892d02d450
[Fominskoe:06235] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)
[0x7f26e6333f20]
[Fominskoe:06235] [ 1] /usr/lib/x86_64-linux-
gnu/libgmp.so.10(__gmpn_copyi+0x4d)[0x7f26e71f5213]
[Fominskoe:06235]

* End of error message *

mpirun noticed that process rank 1 with PID 0 on node Fominskoe exited on signal 11 (Segmentation fault).

I noted that for the last choise mpirun -np 1 main

gives the right answer:

before for rank=0
0
0.1
after for rank=0
0
0.1

    #include "mpi.h"

    #if defined(_DBL)
    typedef double DBL;
    #endif    

    #if defined(_LDBL)
    typedef long double DBL;
    #endif 

    #if defined(_EDBL50)
    #include <boost/multiprecision/gmp.hpp>
    using namespace boost::multiprecision;
    typedef mpf_float_50 DBL;
    #endif 

    using namespace std;

    int main(int argc, char* argv[])
    {

    MPI::Init (argc, argv);
    int proc_num = MPI::COMM_WORLD.Get_size ( );
    int my_rank  = MPI::COMM_WORLD.Get_rank ( );

    int N=2;

    DBL  DB[N];
    int  CN=N*sizeof(DBL);
    char CH[CN];

    if ( !my_rank ){

        cout<<"before for rank="<<my_rank<<endl;
        for (int i=0; i<N; i++) {               // init array
            DB[i]=i*0.1;
            cout<<DB[i]<<endl;
        }

        char* ptr=(char*)(&DB[0]);

        for (int i=0; i<CN; i++) 
            CH[i]=*ptr++;

        for (int i=0; i<N; i++) // clean
            DB[i]=0;

    }

    MPI_Bcast (CH, CN, MPI_CHAR, 0, MPI_COMM_WORLD);

    int ii=0;
    DBL* V;

    cout<<"\nafter for rank="<<my_rank<<endl;
    for (int i=0; i<N; i++) {
        V=(DBL*)(&CH[ii]); 
        DB[i]=*V;
        cout<< DB[i]<<endl;
        ii+=sizeof(DBL);
        //        if (my_rank)
        //        break;
    }

    MPI::Finalize();

    return 0;
}

If MPI is removed conversion DBL->char->DBL works for all three types.

Ubuntu 18.04, gcc, mpicxx -- Open MPI C++ wrapper compiler, libboost-all-dev, libboost-tools-dev

Any ideas?

Solution

  • As the comments say, you can only bit-wise serialize trivial types.

        char *ptr = (char *)(&DB[0]);

    for (int i = 0; i < CN; i++)
        CH[i] = *ptr++;

    This is NOT a conversion. It's a reinterpret cast that achieves a bit-wise copy, That's Undefined Behaviour for the non-trivial multiprecision type.

    You're in luck, since you want to use MPI and Boost, you can use Boost MPI which has built-in support for Boost Serialization, which in turn has built-in support for multiprecision.

    So I'd suggest to use that and spare yourself the head-aches.

    In short, you're not in C-land anymore. If you're using C++ you cannot make the assumptions that C programmers tend to make. That's for the best, because you also don't need to write undefined behaviour anymore, or do all the tedious work manually :)

    Boost MPI Demo:

    #include <boost/mpi.hpp>
#include <boost/serialization/vector.hpp>
#include <boost/multiprecision/cpp_bin_float.hpp>
#include <boost/multiprecision/gmp.hpp>
#include <iostream>
#include <random>

namespace bmp = boost::multiprecision;
namespace mpi = boost::mpi;

//using Database = std::vector<bmp::mpf_float_50>;
using Database = std::vector<bmp::cpp_bin_float_50>;
static std::mt19937 prng { std::random_device{}() };

int main() {
    mpi::environment env;
    mpi::communicator world;

    if (world.rank() == 0) {
        std::cout << "before for rank=" << world.rank() << std::endl;

        Database db;

        std::generate_n(
                back_inserter(db),
                prng()%15, 
                [i=0]() mutable { return i++*0.1; });

        world.send(1, 1, db);
    } else {
        Database db;
        world.recv(0, 1, db);

        std::cout << "Received " << db.size() << " numbers: ";
        for (auto& number : db) {
            std::cout << number << " ";
        }
        std::cout << std::endl;
    }
}

    When run:

    enter image description here

    With Non-Serializable Types

    If you insist on using mpfr types, I think you'll have to do serialization manually. An exceptionally naive way is just converting all the elements to string: 

    #include <boost/mpi.hpp>
#include <boost/serialization/vector.hpp>
#include <boost/multiprecision/cpp_bin_float.hpp>
#include <boost/multiprecision/gmp.hpp>
#include <boost/convert/lexical_cast.hpp>
#include <boost/convert.hpp>
#include <boost/range/algorithm.hpp>
#include <boost/range.hpp>
#include <iostream>
#include <random>

namespace bmp = boost::multiprecision;
namespace mpi = boost::mpi;

using Num = bmp::mpf_float_50;
using Database = std::vector<Num>;
using SerializationFormat = std::vector<std::string>;
static auto serialize   = boost::cnv::apply<std::string>(boost::cnv::lexical_cast());
static auto deserialize = boost::cnv::apply<Num>(boost::cnv::lexical_cast());

//using Database = std::vector<bmp::cpp_bin_float_50>;
static std::mt19937 prng { std::random_device{}() };

int main() {
    mpi::environment env;
    mpi::communicator world;

    if (world.rank() == 0) {
        std::cout << "before for rank=" << world.rank() << std::endl;

        Database db;

        std::generate_n(
                back_inserter(db),
                prng()%15, 
                [i=0]() mutable { return i++*0.1; });

        SerializationFormat db_str;
        boost::transform(db, back_inserter(db_str), serialize);
        world.send(1, 1, db_str);
    } else {
        SerializationFormat db_str;
        world.recv(0, 1, db_str);

        Database db;
        boost::transform(db_str, back_inserter(db), deserialize);

        std::cout << "Received " << db.size() << " numbers: ";
        for (auto& number : db) {
            std::cout << number << " ";
        }
        std::cout << std::endl;
    }
}