Search code examples
rustdllsegmentation-faultrust-cargodangling-pointer

Segfault in Rust when calling a function returned by a cdylib crate function


If I have a fn(&mut Box<fn()>) in a cdylib crate, and it sets the value of the Box to another function defined in the crate, I get a segmentation fault when calling the resulted function.

I have two crates to test this, lib and use-lib. use-lib uses libloading 0.8.6, but there are no other dependencies.

Here is my code in lib/src/lib.rs:

fn dangling() {
    println!("Hello from the returned function");
}

#[no_mangle]
pub extern "Rust" fn loaded_fn(func: &mut Box<fn()>) {
    **func = dangling;
}

Here is my code in use-lib/src/main.rs (I'm loading a DLL because I'm on Windows, I don't know if that affects the problem):

use libloading::{Library, Symbol};

fn default() {
    println!("function not modified");
}

fn main() {
    let mut dangling_fn: Box<fn()> = Box::new(default);

    unsafe {
        type LoadedFn = extern "Rust" fn(&mut Box<fn()>);

        let lib = Library::new("./lib.dll").unwrap_or_else(|err| {
            panic!("unable to load library: {}", err);
        });
        println!("loaded lib");
        
        let loaded_fn: Symbol<LoadedFn> = lib.get(b"loaded_fn").unwrap_or_else(|err| {
            panic!("unable to get symbol: {}", err);
        });
        println!("loaded function");

        loaded_fn(&mut dangling_fn);
        println!("called function");
    }

    println!("calling dangling function");
    dangling_fn();
}

This is the output I get:

loaded lib
loaded function
called function
calling dangling function
error: process didn't exit successfully: `target\debug\use-lib.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

My diagnosis is that returned_fn is never loaded into use_lib, but how can I fix this?


Solution

  • When lib goes out of scope, the library is unloaded, so your dangling_fn is truely dangling after the unsafe block. So you have to lift it out of the scope:

    fn main() {
        let mut dangling_fn: Box<fn()> = Box::new(default);
        let lib;
        unsafe {
            // ...
            lib = Library::new("./lib.dll").unwrap_or_else(|err| {
                panic!("unable to load library: {}", err);
            });
            // ...
        }
    }
    

    Or maybe even better just wrap the parts that actually need unsafe and avoid creating such a big scope to begin with, this also has the added benefit of clearly marking just the unsafe calls:

    use libloading::{Library, Symbol};
    
    fn default() {
        println!("function not modified");
    }
    
    fn main() {
        let mut dangling_fn: Box<fn()> = Box::new(default);
    
        type LoadedFn = extern "Rust" fn(&mut Box<fn()>);
    
        let lib = unsafe { Library::new("./lib.dll") }.unwrap_or_else(|err| {
            panic!("unable to load library: {}", err);
        });
        println!("loaded lib");
    
        let loaded_fn: Symbol<LoadedFn> = unsafe { lib.get(b"loaded_fn") }.unwrap_or_else(|err| {
            panic!("unable to get symbol: {}", err);
        });
        println!("loaded function");
        loaded_fn(&mut dangling_fn);
        println!("called function");
        println!("calling dangling function");
        dangling_fn();
    }