Transmuting from trait A to trait B on type that implements both segfaults

Question

Few days ago, there was a question regarding OOP that wanted to solve the issue using downcast. As a self-imposed challenge, I tried to solve the issue using std::mem::transmute and unsafe blocks, which produced a segmentation fault.

Here is the full code in Rust Playground.

The offending part of code is this:

unsafe {
    let storage =
        mem::transmute::<&mut Box<AnyStorable + 'static>, &mut Box<Insertable<C>>>(x);
    println!("{:#?}", x);
    storage.insert(component); // This segfaults
};

It produces an segfault when run:

/root/entrypoint.sh: line 7: 5 Segmentation fault timeout --signal=KILL ${timeout} "$@"

However, when I replace this line:

let storage =
    mem::transmute::<&mut Box<AnyStorable + 'static>, &mut Box<Insertable<C>>>(x);

with:

let storage =
    mem::transmute::<&mut Box<AnyStorable + 'static>, &mut Box<VecStorage<C>>>(x);

It works. Why does the first line fail and the second doesn't?

Answer 1

Box<SomeTrait> stores two pointers: one to the object, and one to the vtable. Box<SomeType> stores only one pointer: the one to the object.

You can use the following code in your example to see the sizes:

println!("{}", mem::size_of::<Box<AnyStorable + 'static>>());
println!("{}", mem::size_of::<Box<Insertable<C>>>());
println!("{}", mem::size_of::<Box<VecStorage<C>>>());

Calling transmute to change the trait of the Box will break the vtable: vtables of different traits are not compatible.

Calling transmute to change from a reference to Box<SomeTrait> to a reference to Box<SomeType> (and the type happens to be the correct one) happens to works, because it will only use the first pointer to the object and forget about the trait.

The internal representation of a fat pointer (i.e. data pointer with vtable) is defined in TraitObject, which is only accessible in nightly builds. Although unlikely the representation might change in way that the data pointer is not the first pointer anymore, which would break the second transmute.

The documentation for TraitObject is also worth reading.

(While transmute makes sure the sizes of the passed types are equal, you're passing a reference to types - which are always exactly one pointer big. It doesn't check the types those references point to.)