Search code examples
llvmdynamic-linkingexternffillvmlite

Linking to external functions in dynamic libraries with LLVM

In my project, I am emitting LLVM IR which makes calls to external functions in dynamic libraries.

I declare my external functions like:

declare %"my_type"* @"my_function"()

In the external library, functions are declared like:

extern "C" {
    my_type* my_function();
}

When I compile the IR and run it, the process immediately crashes. The same behavior happens if I declare and call a nonsense function that I know doesn't exist, so I assume what's happening is that the external function is not being found/linked. (I don't think that the function itself is crashing).

I am using Python's llvmlite library for this task, and within the same process where I JIT and invoke my LLVM IR, I have another python library imported which requires the external dynamic library; so I assume that library is loaded and in-memory.

The procedure I'm using to compile and execute my LLVM code is basically the same as what's in this document, except that the IR declares and invokes an external function. I have tried invoking cos(), as in the Kaleidoscope tutorial, and this succeeds, so I am not sure what is different about my own library functions.

I have tried adding an underscore to the beginning of the function name, but I get the same result. (Do I need to add the underscore in the LLVM function declaration?)

  • How do I verify my assumption that the process is crashing because the named function isn't found?
  • How do I diagnose why the function isn't being found?
  • What do I need to do in order to make use of external functions in a dynamic library, from LLVM code?

Edit: It seems there is indeed trouble getting the function pointers to my external function. If I try to merely print the function address by replacing my calls with %"foo" = ptrtoint %"my_type"* ()* @"my_function" to i64 and return/print the result, it still segfaults. Merely trying to obtain the pointer is enough to cause a crash! Why is this, and how do I fix it?

Edit: Also forgot to mention— this is on Ubuntu (in a Docker container, on OSX).

Solution

  • I figured it out— I was missing that I need to call llvmlite.binding.load_library_permanently(filename) in order for the external symbols to become available. Even though the library is already in memory, the function still needs to be invoked. (This corresponds to the LLVM native function llvm::sys::DynamicLibrary::LoadLibraryPermanently()).

    From this answer, it appears calling the above function with nullptr will import all the symbols available to the process.

    Oddly, on OSX, I found that the external symbols were available even though load_library_permanently() had not been explicitly called— I'm not sure why this is (perhaps the OSX build of llvmlite itself happened to call the function with nullptr, as above?).