Manually call a rust dynamic library

I'm currently playing around with DynamicLibrary.

The code of my dynamic library (compiled with rustc --crate-type dylib dylib.rs):

// dylib.rs
#[no_mangle]
pub fn minicall() -> u8 {
    3u8
}

And the code to call it:

// caller.rs
use std::dynamic_lib::DynamicLibrary;

fn main() {
    let mut v = Vec::new();

    DynamicLibrary::prepend_search_path(&::std::os::getcwd());

    match DynamicLibrary::open(Some("./libdylib.so")) {
        Err(e) => panic!("ERROR: {}", e),
        Ok(lib) => {
            println!("Unsafe bloc !");
            let func = unsafe {
                match lib.symbol::< fn() -> u8 >("minicall") {
                        Err(e) => { panic!("ERROR: {}", e) },
                        Ok(f) => { *f },
                }
            };
            println!("call func !");
            let new_value = func();

            println!("extend vec !");
            v.push(new_value);
        }
    }

    println!("v is: {}", v);
}

I have this output :

~> ./caller 
Unsafe bloc !
call func !
Illegal instruction

And here I'm quite lost. What am I doing wrong ?

Solution

The problem here is how the symbol function works. It has signature:

unsafe fn symbol<T>(&self, symbol: &str) -> Result<*mut T, String>

A loaded library is basically a big array in memory with certain addresses labelled with a name (the symbol names). Querying for a symbol looks up the address and returns a pointer straight to it. A function in a library is a long sequence of instructions, so querying for a function's name returns a (function) pointer directly to the start. This can then be called as a normal function pointer. The Rust DynamicLibrary API is returning this pointer, that is, *mut T points directly to the chunk of memory in the dynamic library (which is supposedly/hopefully of type T).

The type fn(...) -> ... is a function pointer itself, that is, it is 8 bytes (or 4 bytes) storing the address of the start of the function it represents. Hence, calling lib.symbol::< fn() -> u8 >("minicall") is saying "find me the address of the thing called minicall (which is a pointer to a function)", it is not saying "find me the address of the thing called minicall (which is a function)". The return value of *mut (fn() -> u8) is then doubly-indirect, and dereferencing it to call it is interpreting the first 8 (or 4) bytes of the function code as a pointer (i.e. random machine instructions/function prelude), it is not executing them.

(Side-note: it would probably work if you had #[no_mangle] pub static minicall: fn() -> u8 = the_real_minicall; in your library, but you probably don't want this.)

The call to lib.symbol::<T>("minicall") is returning the exact function pointer we want (that is, it is returning a pointer to the start of the code of minicall), so it just becomes a question of expressing this to the compiler. Unfortunately, there is currently no type T that makes *mut T a function pointer, so one must first set T = u8 (i.e. lib.symbol::<u8>("minicall")) and then cast the return value to the appropriate function pointer type via transmute::<_, fn() -> u8>(pointer).

(I'm answering this even after the other answer was accepted because I don't think it explained the cause very well, just gave the solution.)

Last thing, this isn't a problem in this case, but it trips people a lot: the Rust ABI (the calling convention used for functions of type fn(...) -> ...) is not the same as the C ABI, so functions loaded from C dynamic libraries should be given type extern "C" fn(...) -> ..., not fn(...) -> ....