MSVC: Crash when invoking a method directly via the VTable that returns by value (C++)

Question

I'm trying to understand why the following code does not properly work in MSVC under Windows. It calls a VTable method directly which returns by value.

#include <iostream>

#define RETURN_BY_VALUE 1

struct Value {
  int Secret;
};

class IFoo {
public:
#if RETURN_BY_VALUE
  virtual Value GetValue() = 0;
#else
  virtual int GetValue() = 0;
#endif
};

class Foo : public IFoo {
public:
#if RETURN_BY_VALUE
  virtual Value GetValue() { return Value{42};  }
#else
  virtual int GetValue() { return 42; }
#endif
};

int main() {
  IFoo* f = new Foo;
  void* vtable = (void *)(*(std::intptr_t *)f);
  
#if RETURN_BY_VALUE
  std::cout << f->GetValue().Secret << std::endl;
  
  // The pointer to the GetValue function is at 0 byte offset in the Vtable
  auto method = (Value(*)(IFoo*)) (*(std::intptr_t *)(vtable));  
  std::cout << method(f).Secret << std::endl;
#else
  std::cout << f->GetValue() << std::endl;

  auto method = (int(*)(IFoo*)) (*(std::intptr_t *)(vtable));  
  std::cout << method(f) << std::endl;
#endif
  
  delete f;
  return 0;
}

The output is

> test.exe
42
1820312

If we set RETURN_BY_VALUE to 0 everything works as expected.

> test.exe
42
42

Running the sample in godbolt indicates that gcc and clang do not have this issue https://godbolt.org/z/hn4MxK.

A real world application of this would be hooking GetAdapterLuid in d3d12.

Answer 1

1. Don't forget about __stdcall. Without __stdcall, calling convention for free function and for methods differs more. GetAdapterLuid is __stdcall I guess.
2. With Windows ABI returned structures are passed as hidded out parameter. It looks like that the order between this and the hidden parameter is different in method and free function. You can define your function so that is is really out paramteter.

This seem to work for me, though sure it smells:

#include <iostream>

#define RETURN_BY_VALUE 1

struct Value {
    int Secret;
};

class IFoo {
public:
#if RETURN_BY_VALUE
    virtual Value __stdcall GetValue() = 0;
#else
    virtual int __stdcall  GetValue() = 0;
#endif
};

class Foo : public IFoo {
public:
#if RETURN_BY_VALUE
    virtual Value __stdcall GetValue() { return Value{42};  }
#else
    virtual int __stdcall  GetValue() { return 42; }
#endif
};


int main() {
    IFoo* f = new Foo;
    void* vtable = (void *)(*(std::intptr_t *)f);
  
#if RETURN_BY_VALUE
    std::cout << f->GetValue().Secret << std::endl;
  
    // The pointer to the GetValue function is at 0 byte offset in the Vtable
    Value v;
    auto method = (void(__stdcall *)(IFoo*, Value*)) (*(std::intptr_t *)(vtable));  
    std::cout << (method(f, &v), v.Secret) << std::endl;
#else
    std::cout << f->GetValue() << std::endl;

    auto method = (int(__stdcall *)(IFoo*)) (*(std::intptr_t *)(vtable));  
    std::cout << method(f) << std::endl;
#endif
  
    delete f;
    return 0;
}

---

gcc / clang would return your structure exactly as integer, Unix-like systems have different ABI.

---

The cleanest way to implement hooks is to implement methods as methods. Locate methods in your hook vtable exactly as in target vtable, then you don't need to fake a method using free function.