Search code examples
pythonglibclibctheano

Problems with a local installation of libc

I'm trying to run a Theano implementation of alexNet on some machines at work. When I first tried to run it I got the following error:

$ THEANO_FLAGS=mode=FAST_RUN,floatX=float32 python train.py

 ....

 ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /data1/software/analysis/libstdc++.6.0.15/usr/lib64/libstdc++.so.6)

Normally, what I would do in this situation is use my package manager to update libc, but this is not an option for work/administrative reasons. So, what I did instead was to install a local version of libc and to change LD_LIBRARY_PATH in my .bashrc file to point to it. Now, however, I get the following error:

$ THEANO_FLAGS=mode=FAST_RUN,floatX=float32 python train.py
python: relocation error: /data1/software/analysis/libc.2.14/lib64/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

Oddly enough, I get a similar 'relocation error' when I just try to 'ls'.

Does anyone know how I can install a version of libc that will only be used by my Python interpreter, but not by everything else?

Note: The libstdc++ I'm using is a locally installed version. I installed it for the same reason that I'm trying to install a local version of libc.

...

OK, I've progressed a little further, but am still stuck.

If I return to my old LD_LIBRARY_PATH, in order to avoid errors with commands like 'ls', I can verify that the new (local) libc does, in fact, use the old (system) ld-linux-x86-64.so.2

$ ldd /data1/software/analysis/libc.2.14/lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2 (0x0000003b9d400000)
linux-vdso.so.1 =>  (0x00007fff887ff000)

So, I thought I would change LD_LIBRARY_PATH to look for the new (local) ld-linux-x86-64.so.2

$ export LD_LIBRARY_PATH=/data1/software/analysis/libc.2.14:$LD_LIBRARY_PATH
$ ls /data1/software/analysis/libc.2.14/lib64
  ld-2.14.1.so  libanl-2.14.1.so   libcidn.so.1
  libm-2.14.1.so       libnss_compat.so.2    libnss_hesiod-2.14.1.so 
  libnss_nisplus.so.2   librt-2.14.1.so libutil.so.1
  ld-linux-x86-64.so.2     libanl.so.1        libcrypt-2.14.1.so   
  libm.so.6        libnss_dns-2.14.1.so    libnss_hesiod.so.2
  libpthread-2.14.1.so  librt.so.1      rtkaio
  libBrokenLocale-2.14.1.so  libc-2.14.1.so     libcrypt.so.1     
  libnsl-2.14.1.so     libnss_dns.so.2     libnss_nis-2.14.1.so  
  libpthread.so.0      libthread_db-1.0.so  
  libBrokenLocale.so.1     libc.so.6          libdl-2.14.1.so        
  libnsl.so.1          libnss_files-2.14.1.so  libnss_nis.so.2         
  libresolv-2.14.1.so   libthread_db.so.1
  libSegFault.so           libcidn-2.14.1.so  libdl.so.2      
  libnss_compat-2.14.1.so  libnss_files.so.2       libnss_nisplus-2.14.1.so 
  libresolv.so.2       libutil-2.14.1.so

$ ldd /data1/software/analysis/libc.2.14/lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2 (0x0000003b9d400000)
linux-vdso.so.1 =>  (0x00007fffad5ff000)

$ echo $LD_LIBRARY_PATH
  /data1/software/analysis/libc.2.14:
  /data1/software/analysis    
  /libstdc++.6.0.15/usr/lib64:/data1/software/analysis/anaconda/lib:
  /data2/software/stable/CUDA/5.5.22/lib:
  /data2/software/stable/CUDA/5.5.22/include:
  /data2/software/stable/CUDA/5.5.22/lib64:
  /data1/software/analysis/anaconda/lib:
  /data2/software/stable/CUDA/5.5.22/lib:
  /data2/software/stable/CUDA/5.5.22/include:
  /data2/software/stable/CUDA/5.5.22/lib64:

So, my libc should see the new ld-linux-x86-64.so.2. I can verify that this so has the symbol _dl_starting_up:

$ nm /data1/software/analysis/libc.2.14/lib64/ld-linux-x86-64.so.2 | grep _dl_start*
  0000000000004c00 t _dl_start
  000000000000fc30 t _dl_start_profile 
  0000000000001698 t _dl_start_user
  0000000000220000 V _dl_starting_up
  0000000000220000 b _dl_starting_up_internal

But, when I try to run the alexNet implementation using LD_PRELOAD to guarantee the use of the right libc, I still get the same error:

$ LD_PRELOAD=/data1/software/analysis/libc.2.14/lib64/libc.so.6 THEANO_FLAGS=mode=FAST_RUN,floatX=float32 python train.py

python: relocation error: /data1/software/analysis/libc.2.14/lib64/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

Why is the new (local) ld-linux not overriding the old (system) ld-linux? Shouldn't my setting of LD_LIBRARY_PATH have taken care of that?

If I try to force the issue, by preloading ld-linux, I get a segmentation fault:

$ LD_PRELOAD=/data1/software/analysis/libc.2.14/lib64/ld-linux-x86-64.so.2:/data1/software/analysis/libc.2.14/lib64/libc.so.6 THEANO_FLAGS=mode=FAST_RUN,floatX=float32 python train.py
Segmentation fault (core dumped)

So, now I'm stuck. I don't know

  1. Why my setting of LD_LIBRARY_PATH didn't force my new (local) libc to use the new (local) ld-linux.

  2. What I should do about the segmentation fault when I force libc to use the new (local) ld-linux.

Solution

  • So, what I did instead was to install a local version of libc and to change LD_LIBRARY_PATH in my .bashrc file to point to it

    See this answer for why that can't work.

    Why my setting of LD_LIBRARY_PATH didn't force my new (local) libc to use the new (local) ld-linux.

    As above answer explains, ld-linux is baked into your executable at link time and can't be changed by LD_LIBRARY_PATH.

    What I should do about the segmentation fault when I force libc to use the new (local) ld-linux.

    LD_PRELOADing ld-linux can never work: LD_PRELOAD is interpreted by ld-linux, so you are effectively forcing two separate ld-linuxes into your process, and that greatly confuses both of them.

    So, what can you do?

    It is in fact possible to install two separate versions of glibc on a single host, we do that every day. Instructions here.