Search code examples
c++linuxdockerdebian

Why does loading some custom libraries fail?

I compiled several shared libraries with my own code and added the -m32 option during compilation. When I used the following code (dlopen) function to load it, threw this exception.

#1 exception:

terminate called after throwing an instance of 'LibraryLoadException'
  what():  shared_library.cpp : 39 : lib/libCR_TcpControl.so: cannot allocate memory in static TLS block

#2 code:

void SharedLibrary::load(bool initial)
{
    if (handle_)
        throw LibraryAlreadyLoadedException(path_);

    handle_ = dlopen(path_.c_str(), RTLD_NOW | RTLD_DEEPBIND);

    LOG("dlopen : {} : {} ", handle_, path_.c_str());

    if (!handle_) {
        const char* err = dlerror();
        throw LibraryLoadException(err ? std::string(err) : path_);
    }

    reinterpret_cast<ModuleInitFunc>(getSymbol("moduleInit"))();

    if (initial) {
        std::string module_name = reinterpret_cast<GetModuleNameFunc>(getSymbol("getModuleName"))();
        ModuleManager::instance()->initModule(module_name);
    }

    if (watch_)
        file_watch_->addWatch(path_, std::bind(&SharedLibrary::watchEvent, this));
}

I have seen from other places that preload can be used to solve this problem. I followed suit, but there will be another warning as follows:

#3 warning:

ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/bin/sh: 1: wget: not found
ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
file /qemu.sh is not exist ,will not run it .
file /develop.sh is not exist ,will not run it .
dobot_simulate start done !!!!
root@7c6404f6b677:/dobot/bin# this stream will redirect to channel1
ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/bin/sh: 1: kill: No such process

ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/dobot/bin/lib/libCR_TcpControl.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

I have checked some information and said that it is because my system is 64 bit, but my library is 32 bit, so this warning is reported. Then I'm even more confused, why would I have no problem loading other libraries (such as libCR_RobotFSM.so), and only this libCR_TcpControl.so would have problems.

#4 the information of file:

root@7c6404f6b677:/dobot/bin# file lib/libCR_RobotFSM.so
lib/libCR_RobotFSM.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, not stripped
root@7c6404f6b677:/dobot/bin# file lib/libCR_TcpControl.so
lib/libCR_TcpControl.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (GNU/Linux), dynamically linked, not stripped

By the way, my program runs in the Docker container, using debian:bookworm for mirroring, and the system is indeed 64 bit.

#5 dockerfile:

#FROM debian:latest
FROM debian:bookworm
# for 32bit
RUN dpkg --add-architecture i386

RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install net-tools \
    uml-utilities \
    bridge-utils\
    iputils-ping \
    python3 \
    vim \
    openssh-server \
    openssh-client \
    curl \
    gpg \
    psmisc \
    samba \
    samba-common \
    redis \
    unzip \
    file \
    libc-bin \
    -y

RUN apt-get install libc6:i386 -y
RUN apt-get install lib32stdc++6 -y
Solution

  • LD_PRELOAD will generally not solve this specific issue, it will in fact make it worse. (The issue you are describing actually occurs more often when LD_PRELOAD is used, which is why you may find lots of references to that in this context.)

    The problem is that the libraries you are loading via dlopen() use thread local storage (i.e. have variables that are declared with thread_local (C++ standard) or __thread (GNU extension), e.g. thread_local static int x;. There are various different ways the compiler can implement these, and this can be specified via -ftls-model=XXX. The issues is that the compiler will likely try to choose the model initial-exec if it thinks that's possible, which is completely fine if you are building an executable or a library that is always loaded when the executable starts, but is an issue with plugins. (But it makes accessing the TLS variables a lot faster, which is why the compiler tries to optimize in that manner.) The problem is that the initial-exec model requires space for each variable in an internal vector when the program is executed, and when a program is started, the size of that vector is determined by the number of TLS variables in the executable + all of its dependencies + some additional surplus space. (It's actually the total size of the variables and not the number of them, but that doesn't help you.)

    When you load a plugin that uses a TLS model that requires space in that vector, the dynamic loader will start using up the surplus. But once that's full, you will get the error message you are seeing.

    If you control the plugin you are loading, you can compile the plugin (or potentially other plugins you are also loading from the program) with the compiler option -ftls-model=global-dynamic to switch to non-static thread-local storage.

    However, if your plugin itself is linked against other libraries and those use static TLS, you will still run into the same problem. For example, if you dlopen() a plugin called plugin_FANCY.so, but that plugin is itself linked against a libFANCY.so, and libFANCY.so has the TLS variables, then you'd also need to recompile libFANCY.so with the -ftls-model=global-dynamic compiler flag (if that's possible for you).

    If you can't change enough of the libraries you are loading yourself to use -ftls-model=global-dynamic, then you're kind of out of luck. You could theoretically recompile the system C library (glibc) and increase the amount of surplus space it allocates for TLS, but I really would recommend against such a thing.

    Alternatively, if dependencies of the plugins are an issue, and those can't be recompiled by you, you could just link your executable directly against at least some of those libraries, so that the initial size of the static TLS area at program start is larger and already includes most variables in question. (That kind of defeats the purpose of loading plugins to some extent though.)

    Further advanced reading on this issue: