I have a code which was dynamically linked with the mkl library. when running the code, it reports mkl not found.
./bmdl
/g/software/EMTO/5.7/intel_12.1/ser/bin/bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
But when I use ldd to check the dynamically linked libraries in the executable, it shows the mkl library is found
ldd bmdl
libmkl_intel_lp64.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_intel_lp64.so (0x00002b975d76d000)
libmkl_sequential.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_sequential.so (0x00002b975df53000)
libmkl_core.so => /g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/libmkl_core.so (0x00002b975e631000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003785600000)
libm.so.6 => /lib64/libm.so.6 (0x0000003784e00000)
libc.so.6 => /lib64/libc.so.6 (0x0000003784a00000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000378a600000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003785200000)
/lib64/ld-linux-x86-64.so.2 (0x0000003784600000)
Any idea what could be wrong?
Output from readelf -l ./bmdl
Elf file type is EXEC (Executable file)
Entry point 0x4034b0
There are 8 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000400040 0x0000000000400040
0x00000000000001c0 0x00000000000001c0 R E 8
INTERP 0x0000000000000200 0x0000000000400200 0x0000000000400200
0x000000000000001c 0x000000000000001c R 1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x0000000000000000 0x0000000000400000 0x0000000000400000
0x00000000000e4eb4 0x00000000000e4eb4 R E 200000
....
More debugging
$ export LD_DEBUG=libs,files
$ ./bmdl
./bmdl: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
$ ldd ./bmdl
15133:
15133: file=libtermcap.so.2 [0]; needed by /bin/sh [0]
15133: find library=libtermcap.so.2 [0]; searching
15133: search path=/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/mkl/lib/intel64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/tls:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64/x86_64:/g/software/intelXE/composer_xe_2011_sp1/lib/intel64 (LD_LIBRARY_PATH)
....
Seems LD_DEBUG doesn't have effect on running ./bmdl alone.
I just realized the old bmdl has a 'setgid' flag and my new built doesn't have it. Maybe that was the cause?
-rwxr-sr-x 1 root gants 1123992 Jul 23 16:14 /scratch/helpdesk/bmdl
I removed the setgid bit from the old bmdl and running ./bmdl doesn't complain about library not found. Now the question is why setgid can interfere with dynamically linked library?
It happens that setgid with a dynamically linked executable can cause security problems and is greatly limited by GNU glibc. For example, LD_LIBRARY_PATH will be ignored. Maybe the old built never worked before?!
The code is dynamically linked and has the setgid attribute. setgid with a dynamically linked executable can cause security problems and is greatly limited by GNU glibc. For example, LD_LIBRARY_PATH will be ignored. That is why the code keeps on complaining some shared library not found.