A newbie question about shared library: In C, when loading a dynamic library, we use dlopen and then dlsym to find a symbol or a function. Now let say, the function we look for in the dll is typed as:
int add(int a, int b);
but if we cast it to another type, say,
typedef int (*sum)(int a, int b, int c);
what will happen? Will C runtime complain about it?
Thanks!
Long story short, this is undefined behaviour. Calling a function with the wrong number of parameters in C (which you can do by casting function pointers even without dlopen/dlfree) produces undefined behaviour.
For callee-clean calling conventions, like stdcall, using the wrong number or type of arguments will produce a stack imbalance (the callee adjusts the stack to the wrong place). A stack imbalance will quickly kill the program (if you're lucky) by corrupting the return address and local variables of the caller.
For caller-clean calling conventions, the effect is much like calling e.g. printf with the wrong number of arguments: the function may behave erratically by using garbage arguments, but your program might not blow up. (This in general is not desirable; an attacker could exploit this to take control of your program, for example).