I have a local build of glibc.
When I run python from the command-line with it, it is successful:
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64:$HOME/my_libc_env/lib/x86_64-linux-gnu/:$HOME/my_libc_env/usr/lib64/:$HOME/my_libc_env/usr/lib/" $HOME/my_libc_env/lib/x86_64-linux-gnu/ld-2.17.so $HOME/anaconda2/envs/tf_011g/bin/python2.7
When I wrap it inside a bash script, and call it from the command line, it is also successful. (I also replaced the python symbolic link with the bash script)
#!/bin/bash
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64:$HOME/my_libc_env/lib/x86_64-linux-gnu/:$HOME/my_libc_env/usr/lib64/:$HOME/my_libc_env/usr/lib/" $HOME/my_libc_env/lib/x86_64-linux-gnu/ld-2.17.so $HOME/anaconda2/envs/tf_011g/bin/python2.7 "${@}"
However, when my IDE debugger calls the bash script, I get Segmentation Fault.
Running it with LD_DEBUG=files,libs yields the output below.
Any idea how to fix that?
5705:
5705: file=libtinfo.so.5 [0]; needed by /bin/bash [0]
5705: find library=libtinfo.so.5 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libtinfo.so.5
5705:
5705: file=libtinfo.so.5 [0]; generating link map
5705: dynamic: 0x000000308fc1f1e0 base: 0x0000000000000000 size: 0x0000000000220260
5705: entry: 0x000000308fa0c840 phdr: 0x000000308fa00040 phnum: 6
5705:
5705:
5705: file=libdl.so.2 [0]; needed by /bin/bash [0]
5705: find library=libdl.so.2 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libdl.so.2
5705:
5705: file=libdl.so.2 [0]; generating link map
5705: dynamic: 0x0000003f8d802da0 base: 0x0000000000000000 size: 0x0000000000203100
5705: entry: 0x0000003f8d600de0 phdr: 0x0000003f8d600040 phnum: 9
5705:
5705:
5705: file=libc.so.6 [0]; needed by /bin/bash [0]
5705: find library=libc.so.6 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libc.so.6
5705:
5705: file=libc.so.6 [0]; generating link map
5705: dynamic: 0x0000003f8dd8db40 base: 0x0000000000000000 size: 0x0000000000393928
5705: entry: 0x0000003f8da1ee30 phdr: 0x0000003f8da00040 phnum: 10
5705:
5705:
5705: prelink checking: ok
5705:
5705: calling init: /lib64/libc.so.6
5705:
5705:
5705: calling init: /lib64/libdl.so.2
5705:
5705:
5705: calling init: /lib64/libtinfo.so.5
5705:
5705:
5705: initialize program: /bin/bash
5705:
5705:
5705: transferring control: /bin/bash
5705:
5705:
5705: file=libnss_files.so.2 [0]; needed by /bin/bash [0]
5705: find library=libnss_files.so.2 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libnss_files.so.2
5705:
5705: file=libnss_files.so.2 [0]; generating link map
5705: dynamic: 0x00007f5963ec5df8 base: 0x00007f5963cb9000 size: 0x000000000020d590
5705: entry: 0x00007f5963cbb1f0 phdr: 0x00007f5963cb9040 phnum: 9
5705:
5705:
5705: calling init: /lib64/libnss_files.so.2
5705:
5705: opening file=/lib64/libnss_files.so.2 [0]; direct_opencount=1
5705:
5705:
5705: file=libnss_nis.so.2 [0]; needed by /bin/bash [0]
5705: find library=libnss_nis.so.2 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libnss_nis.so.2
5705:
5705: file=libnss_nis.so.2 [0]; generating link map
5705: dynamic: 0x00007f5963cb7de0 base: 0x00007f5963aae000 size: 0x000000000020a4b8
5705: entry: 0x00007f5963ab0020 phdr: 0x00007f5963aae040 phnum: 9
5705:
5705:
5705: file=libnsl.so.1 [0]; needed by /lib64/libnss_nis.so.2 [0]
5705: find library=libnsl.so.1 [0]; searching
5705: search cache=/etc/ld.so.cache
5705: trying file=/lib64/libnsl.so.1
5705:
5705: file=libnsl.so.1 [0]; generating link map
5705: dynamic: 0x0000003f9e015da8 base: 0x0000000000000000 size: 0x0000000000218ab0
5705: entry: 0x0000003f9de04070 phdr: 0x0000003f9de00040 phnum: 9
5705:
5705:
5705: calling init: /lib64/libnsl.so.1
5705:
5705:
5705: calling init: /lib64/libnss_nis.so.2
5705:
5705: opening file=/lib64/libnss_nis.so.2 [0]; direct_opencount=1
5705:
5706: file=/home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]; generating link map
5706: dynamic: 0x00000000006009c0 base: 0x0000000000000000 size: 0x0000000000200bf8
5706: entry: 0x0000000000400700 phdr: 0x0000000000400040 phnum: 8
5706:
5706:
5706: file=libpython2.7.so.1.0 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libpython2.7.so.1.0 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib/tls/x86_64:/home/user/anaconda2/envs/tf_011g/bin/../lib/tls:/home/user/anaconda2/envs/tf_011g/bin/../lib/x86_64:/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/tls/x86_64/libpython2.7.so.1.0
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/tls/libpython2.7.so.1.0
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/x86_64/libpython2.7.so.1.0
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libpython2.7.so.1.0
5706:
5706: file=libpython2.7.so.1.0 [0]; generating link map
5706: dynamic: 0x00007fa7bcfd2c68 base: 0x00007fa7bcc37000 size: 0x00000000003fb4e0
5706: entry: 0x00007fa7bcc78250 phdr: 0x00007fa7bcc37040 phnum: 5
5706:
5706:
5706: file=libpthread.so.0 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libpthread.so.0 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libpthread.so.0
5706: search path=tls/x86_64:tls:x86_64::/usr/local/cuda-7.5/lib64/tls/x86_64:/usr/local/cuda-7.5/lib64/tls:/usr/local/cuda-7.5/lib64/x86_64:/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64/tls/x86_64:/usr/local/cuda-8.0/lib64/cuda/lib64/tls:/usr/local/cuda-8.0/lib64/cuda/lib64/x86_64:/usr/local/cuda-8.0/lib64/cuda/lib64:/my_libc_env/lib/x86_64-linux-gnu/tls/x86_64:/my_libc_env/lib/x86_64-linux-gnu/tls:/my_libc_env/lib/x86_64-linux-gnu/x86_64:/my_libc_env/lib/x86_64-linux-gnu:/my_libc_env/usr/lib64/tls/x86_64:/my_libc_env/usr/lib64/tls:/my_libc_env/usr/lib64/x86_64:/my_libc_env/usr/lib64:/my_libc_env/usr/lib/tls/x86_64:/my_libc_env/usr/lib/tls:/my_libc_env/usr/lib/x86_64:/my_libc_env/usr/lib (LD_LIBRARY_PATH)
5706: trying file=tls/x86_64/libpthread.so.0
5706: trying file=tls/libpthread.so.0
5706: trying file=x86_64/libpthread.so.0
5706: trying file=libpthread.so.0
5706: trying file=/usr/local/cuda-7.5/lib64/tls/x86_64/libpthread.so.0
5706: trying file=/usr/local/cuda-7.5/lib64/tls/libpthread.so.0
5706: trying file=/usr/local/cuda-7.5/lib64/x86_64/libpthread.so.0
5706: trying file=/usr/local/cuda-7.5/lib64/libpthread.so.0
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/tls/x86_64/libpthread.so.0
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/tls/libpthread.so.0
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/x86_64/libpthread.so.0
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/tls/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/tls/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib64/tls/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib64/tls/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib64/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib64/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib/tls/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib/tls/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/usr/lib/libpthread.so.0
5706: search cache=/etc/ld.so.cache
5706: trying file=/lib64/libpthread.so.0
5706:
5706: file=libpthread.so.0 [0]; generating link map
5706: dynamic: 0x0000003f8e017d90 base: 0x0000000000000000 size: 0x000000000021c3f0
5706: entry: 0x0000003f8de05e30 phdr: 0x0000003f8de00040 phnum: 9
5706:
5706:
5706: file=libdl.so.2 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libdl.so.2 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libdl.so.2
5706: search path=tls/x86_64:tls:x86_64::/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64 (LD_LIBRARY_PATH)
5706: trying file=tls/x86_64/libdl.so.2
5706: trying file=tls/libdl.so.2
5706: trying file=x86_64/libdl.so.2
5706: trying file=libdl.so.2
5706: trying file=/usr/local/cuda-7.5/lib64/libdl.so.2
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/libdl.so.2
5706: search cache=/etc/ld.so.cache
5706: trying file=/lib64/libdl.so.2
5706:
5706: file=libdl.so.2 [0]; generating link map
5706: dynamic: 0x0000003f8d802da0 base: 0x0000000000000000 size: 0x0000000000203100
5706: entry: 0x0000003f8d600de0 phdr: 0x0000003f8d600040 phnum: 9
5706:
5706:
5706: file=libutil.so.1 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libutil.so.1 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libutil.so.1
5706: search path=tls/x86_64:tls:x86_64::/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64 (LD_LIBRARY_PATH)
5706: trying file=tls/x86_64/libutil.so.1
5706: trying file=tls/libutil.so.1
5706: trying file=x86_64/libutil.so.1
5706: trying file=libutil.so.1
5706: trying file=/usr/local/cuda-7.5/lib64/libutil.so.1
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/libutil.so.1
5706: search cache=/etc/ld.so.cache
5706: trying file=/lib64/libutil.so.1
5706:
5706: file=libutil.so.1 [0]; generating link map
5706: dynamic: 0x0000003f9fe01e10 base: 0x0000000000000000 size: 0x0000000000202100
5706: entry: 0x0000003f9fc00e10 phdr: 0x0000003f9fc00040 phnum: 9
5706:
5706:
5706: file=libm.so.6 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libm.so.6 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libm.so.6
5706: search path=tls/x86_64:tls:x86_64::/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64 (LD_LIBRARY_PATH)
5706: trying file=tls/x86_64/libm.so.6
5706: trying file=tls/libm.so.6
5706: trying file=x86_64/libm.so.6
5706: trying file=libm.so.6
5706: trying file=/usr/local/cuda-7.5/lib64/libm.so.6
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/libm.so.6
5706: search cache=/etc/ld.so.cache
5706: trying file=/lib64/libm.so.6
5706:
5706: file=libm.so.6 [0]; generating link map
5706: dynamic: 0x0000003f8e882df8 base: 0x0000000000000000 size: 0x00000000002830b8
5706: entry: 0x0000003f8e603e70 phdr: 0x0000003f8e600040 phnum: 9
5706:
5706:
5706: file=libc.so.6 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7 [0]
5706: find library=libc.so.6 [0]; searching
5706: search path=/home/user/anaconda2/envs/tf_011g/bin/../lib (RPATH from file /home/user/anaconda2/envs/tf_011g/bin/python2.7)
5706: trying file=/home/user/anaconda2/envs/tf_011g/bin/../lib/libc.so.6
5706: search path=tls/x86_64:tls:x86_64::/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64 (LD_LIBRARY_PATH)
5706: trying file=tls/x86_64/libc.so.6
5706: trying file=tls/libc.so.6
5706: trying file=x86_64/libc.so.6
5706: trying file=libc.so.6
5706: trying file=/usr/local/cuda-7.5/lib64/libc.so.6
5706: trying file=/usr/local/cuda-8.0/lib64/cuda/lib64/libc.so.6
5706: search cache=/etc/ld.so.cache
5706: trying file=/lib64/libc.so.6
5706:
5706: file=libc.so.6 [0]; generating link map
5706: dynamic: 0x0000003f8dd8db40 base: 0x0000000000000000 size: 0x0000000000393928
5706: entry: 0x0000003f8da1ee30 phdr: 0x0000003f8da00040 phnum: 10
5706:
5706:
5706: calling init: /lib64/libpthread.so.0
5706:
5706:
5706: calling init: /home/user/my_libc_env/lib/x86_64-linux-gnu/ld-2.17.so
5706:
5706:
5706: calling init: /lib64/libc.so.6
5706:
/home/user/anaconda2/envs/tf_011g/bin/python: line 12: 5706 Segmentation fault (core dumped) LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-7.5/lib64:/usr/local/cuda-8.0/lib64/cuda/lib64:$HOME/my_libc_env/lib/x86_64-linux-gnu/:$HOME/my_libc_env/usr/lib64/:$HOME/my_libc_env/usr/lib/" /home/user/my_libc_env/lib/x86_64-linux-gnu/ld-2.17.so /home/user/anaconda2/envs/tf_011g/bin/python2.7 "${@}"
5705:
5705: calling fini: /bin/bash [0]
5705:
5705:
5705: calling fini: /lib64/libtinfo.so.5 [0]
5705:
5705:
5705: calling fini: /lib64/libdl.so.2 [0]
5705:
5705:
5705: calling fini: /lib64/libnss_nis.so.2 [0]
5705:
5705:
5705: calling fini: /lib64/libnss_files.so.2 [0]
5705:
5705:
5705: calling fini: /lib64/libnsl.so.1 [0]
5705:
5705:
5705: calling fini: /lib64/libc.so.6 [0]
5705:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Your shell script contains:
LD_LIBRARY_PATH="...:$HOME/my_libc_env/lib/x86_64-linux-gnu/:..."
and
$HOME/anaconda2/envs/tf_011g/bin/python2.7
Your trace contains:
file=libpython2.7.so.1.0 [0]; needed by /home/user/anaconda2/envs/tf_011g/bin/python2.7
and
trying file=/my_libc_env/lib/x86_64-linux-gnu/tls/x86_64/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/tls/libpthread.so.0
5706: trying file=/my_libc_env/lib/x86_64-linux-gnu/x86_64/libpthread.so.0
It appears that one expansion of $HOME produced /home/user, while another expansion of $HOME in the same shell script produced empty string.
That seems to be exceedingly unlikely to happen, so I suspect you've actually edited one or the other, and aren't telling us the whole truth, and nothing but the truth.
Anyway, it seems possible for $HOME to not be set when running from within IDE, and that would certainly explain both the search path observed, and the crash (the binary ended up loading /lib64/libpthread.so.0, which is not compatible with the ld-linux you are running it with).