Search code examples
ccmakempiscalapack

Use of MPI::MPI_C alias

I am building ScaLAPACK (commit 7e4e07070a489686287c36ab473d21cf29a54bdd) using CMake (version 3.20.1). The find_package utility is used in CMakeLists.txt to define the libraries, inclusion directories, and flags required to compile with MPI. The ScaLAPACK library is linked with MPI using the alias MPI::MPI_C:

target_link_libraries( scalapack ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES} MPI::MPI_C)

I am building a shared library, so according to CMake default behavior, the resulting DSO should not have any entries in its RPATH. However, the installed library DSO maintains in its RUNPATH the entries:

$ readelf -d ~/opt/scalapack/lib/libscalapack.so | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/opt/apps/resif/aion/2020b/epyc/software/UCX/1.9.0-GCCcore-10.2.0/lib64:/usr/lib64:/opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib]

If the library is linked with using the libraries variable MPI_C_LIBRARIES defined by the find_package utility,

target_link_libraries( scalapack ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES} ${MPI_C_LIBRARIES})

then there is no RUNPATH entry in the installed DSO.

Looking further into the Makefiles generated by CMake, I noticed the following differences:

  • With alias MPI::MPI_C:
# File: <build directory>/CMakeFiles/scalapack.dir/DependInfo.cmake
...
set(CMAKE_Fortran_TARGET_INCLUDE_PATH
  "/mnt/irisgpfs/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/include"
  "/opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/include"
)
...

# File: <build directory>/CMakeFiles/scalapack.dir/Dflags.make
...
C_INCLUDES = -isystem /opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/include

C_FLAGS = -O3 -DNDEBUG -fPIC -fexceptions

Fortran_DEFINES = -DAdd_ -Dscalapack_EXPORTS

Fortran_INCLUDES = -I/mnt/irisgpfs/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/include -I/opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/include

Fortran_FLAGS = -fallow-argument-mismatch -O2 -DNDEBUG -O2 -fPIC -fexceptions
...


# File: <build directory>/CMakeFiles/scalapack.dir/link.txt

/opt/apps/resif/aion/2020b/epyc/software/GCCcore/10.2.0/bin/gfortran -fPIC -fallow-argument-mismatch -O2 -DNDEBUG -O2 -Wl,-rpath -Wl,/opt/apps/resif/aion/2020b/epyc/software/UCX/1.9.0-GCCcore-10.2.0/lib64 -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib -Wl,--enable-new-dtags -L/mnt/irisgpfs/apps/resif/aion/2020b/epyc/software/UCX/1.9.0-GCCcore-10.2.0/lib -L/usr/lib64 -L/mnt/irisgpfs/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib -shared -Wl,-soname,libscalapack.so.2.2 -o lib/libscalapack.so.2.2.1 @CMakeFiles/scalapack.dir/objects1.rsp  -Wl,-rpath,/home/users/gkafanas/opt/openblas/lib64: /home/users/gkafanas/opt/openblas/lib64/libopenblas.so -lpthread -lm -ldl /home/users/gkafanas/opt/openblas/lib64/libopenblas.so -lpthread -lm -ldl /opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib/libmpi.so
  • With variable MPI_C_LIBRARIES:
# File: <build directory>/CMakeFiles/scalapack.dir/DependInfo.cmake
...
set(CMAKE_Fortran_TARGET_INCLUDE_PATH
)
...

# File: <build directory>/CMakeFiles/scalapack.dir/Dflags.make
...
C_INCLUDES = 

C_FLAGS = -O3 -DNDEBUG -fPIC

Fortran_DEFINES = -DAdd_ -Dscalapack_EXPORTS

Fortran_INCLUDES = 

Fortran_FLAGS = -fallow-argument-mismatch -O2 -DNDEBUG -O2 -fPIC
...


# File: <build directory>/CMakeFiles/scalapack.dir/link.txt

/opt/apps/resif/aion/2020b/epyc/software/GCCcore/10.2.0/bin/gfortran -fPIC -fallow-argument-mismatch -O2 -DNDEBUG -O2 -shared -Wl,-soname,libscalapack.so.2.2 -o lib/libscalapack.so.2.2.1 @CMakeFiles/scalapack.dir/objects1.rsp  -Wl,-rpath,/home/users/gkafanas/opt/openblas/lib64: /home/users/gkafanas/opt/openblas/lib64/libopenblas.so -lpthread -lm -ldl /home/users/gkafanas/opt/openblas/lib64/libopenblas.so /opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib/libmpi.so -lpthread -lm -ldl /opt/apps/resif/aion/2020b/epyc/software/OpenMPI/4.0.5-GCC-10.2.0/lib/libmpi.so

So it seems that the alias sets some system options that are not necessary.

I have 2 questions:

  1. What is the alias MPI::MPI_C and how it should be used.
  2. Why the alias sets so may parameters when someone might expect that it will set only the variables used to link the ScaLAPACK DSO with MPI.
Solution

  • I'll answer your question indirectly. Since CMake is so little standardized, I have no idea what MPI::MPI_C is or whether it can be made to work. Here is what I use:

    find_package( MPI )

add_executable( ${PROJECT_NAME} ${PROJECT_NAME}.c )
target_include_directories(
        ${PROJECT_NAME} PUBLIC
        ${MPI_C_INCLUDE_DIRS} ${CMAKE_CURRENT_SOURCE_DIR} )
target_link_libraries(
        ${PROJECT_NAME} PUBLIC
        ${MPI_C_LIBRARIES} )

    This is for a single source file, but it can be adapted to your own uses.