Check whether function called through function-pointer has a return statement

Question

We have a plugin system that calls functions in dlls (user-generated plugins) by dlopening/LoadLibrarying the dll/so/dylib and then dlsyming/GetProcAddressing the function, and then storing that result in a function pointer.

Unfortunately, due to some bad example code being copy-pasted, some of these dlls in the wild do not have the correct function signature, and do not contain a return statement.

A dll might contain this:

extern "C" void Foo() { stuffWithNoReturn(); } // copy-paste from bad code

or it might contain this:

extern "C" int Foo() { doStuff(); return 1; } // good code

The application that loads the dll relies on the return value, but there are a nontrivial number of dlls out there that don't have the return statement. I am trying to detect this situation, and warn the user about the problem with his plugin.

This naive code should explain what I'm trying to do:

typedef int (*Foo_f)(void);
Foo_f func = (Foo_f)getFromDll(); // does dlsym or GetProcAddress depending on platform
int canary = 0x42424242;
canary = (*func)();
if (canary == 0x42424242)
   printf("You idiot, this is the wrong signature!!!\n");
else
   real_return_value = canary;

This unfortunately does not work, canary contains a random value after calling a dll that has the known defect. I naively assumed calling a function with no return statement would leave the canary intact, but it doesn't.

My next idea was to write a little bit of inline assembler to call the function, and check the eax register upon return, but Visual Studio 2015 doesn't allow __asm() in x64 code anymore.

I know there is no standards-conform solution to this, as casting the function pointer to the wrong type is of course undefined behavior. But if someone has a solution that works at least on 64bit Windows with Visual C++, or a solution that works with clang on MacOS, I would be most delighted.

Answer 1

@Lorinczy Zsigmond is right in that the contents of the register are undefined if the function does something but returns nothing. We found however that in practice, the plugins that return nothing also have almost always empty functions that compile to a retn 0x0 and leaves the return register untouched. We can detect this case by spraying the rax register with a known value (0xdeadbeef) and checking for that.