Search code examples
pythonfortrancythonhdf5cray

H5py installation via setup.py, undefined symbol: iso_c_binding_

I'm installing the h5py according to the tutorial at http://docs.h5py.org/en/latest/build.html The installation is sucessfull. However, the test failed, 

python setup.py test

I got this:

running test
running build_py
running build_ext
Summary of the h5py configuration
Path to HDF5: '/opt/cray/hdf5-parallel/1.8.13/cray/83/'
HDF5 Version: '1.8.13'
MPI Enabled: True
Rebuild Required: False

Executing cythonize()
Traceback (most recent call last):
File "setup.py", line 140, in <module>
cmdclass = CMDCLASS,
File "/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in setup    
dist.run_commands()
File "/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands    
self.run_command(cmd)
File "/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command    
cmd_obj.run()
File "setup.py", line 68, in run 
import h5py
File "/h5py-2.5.0/build/lib.linux-x86_64-2.7/h5py/__init__.py", line 13, in <module>
from . import _errors
**ImportError:** /opt/cray/lib64/libmpichf90_cray.so.3: undefined symbol: iso_c_binding_

looks like cython cant not find the shared library, how can I attach that? Thanks.

Solution

  • (Edited for parallel build)

    I got this to work on a Cray XC30 (ARCHER: http://www.archer.ac.uk) using the following:

    module swap PrgEnv-cray PrgEnv-gnu
module load cray-hdf5-parallel
export CRAYPE_LINK_TYPE=dynamic
export CC=cc

    ARCHER has specific modules for the Python enviroment on the compute nodes that link to performant versions of numpy etc. (see: http://www.archer.ac.uk/documentation/user-guide/python.php) so I also loaded these (this may not apply on your Cray system, in ARCHER's case mpi4py is already inlcuded in the python-compute install):

    module load python-compute
module load pc-numpy

    Finally, I added the custom install location I will use for h5py to PYTHONPATH

    export PYTHONPATH=/path/to/h5py/install/lib/python2.7/site-packages:$PYTHONPATH

    Now I can build:

    python setup.py configure --mpi
python setup.py install --prefix=/path/to/h5py/install
...lots of output...

    Now, running the tests on the frontend node fail but with the error message I would expect to see on a Cray XC if you try to launch MPI code on a login/service node (failed to initialize communication channel, the login/service nodes are not connected to the high performance network so cannot run MPI code). This suggests to me that the test would probably work if it was running on the compute nodes.

    > python setup.py test
running test
running build_py
running build_ext
Autodetected HDF5 1.8.13
********************************************************************************
                   Summary of the h5py configuration

Path to HDF5: '/opt/cray/hdf5-parallel/1.8.13/GNU/49'
HDF5 Version: '1.8.13'
 MPI Enabled: True
Rebuild Required: False

********************************************************************************
Executing cythonize()
[Thu Oct 22 19:53:01 2015] [unknown] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(547): 
MPID_Init(203).......: channel initialization failed
MPID_Init(579).......:  PMI2 init failed: 1 
Aborted

    To test properly you would have to submit a job that launched a parallel Python script on the compute nodes using aprun. I do not think the built in test framework will work easily as it probably expects the MPI launcher to be called mpiexec (as on a standard cluster) so you may need to write your own tests. The other option would be to coerce setup.py to use aprun instead somehow.