Imagine you'd like to write a program that tests functions in a c++ dll file. You should enable the user to select a dll (we assume we are talking about c++ dlls). He should be able to obtain a list of all functions exported by the dll. Then, the user should be able to select a function name from the list, manually input a list of arguments ( the arguments are all basic types, like int, double, bool or char arrays (e.g. c-type strings) ) and attempt to run the selected function with the specified arguments. He'd like to know if the function runs with the specified arguments, or do they cause it to crash ( because they don't match the signature for example ).
The main problem is that C++, being a strongly typed language, requires you to know the number and type of the arguments for a function call at compile time.And in my case, I simply don't know what these arguments are, until the user selects them at runtime.
The only solution I came up with, was to use assembly to manually push the arguments on the call stack.
However, I've come to understand that if I want to mess with assembly, I'd better make damn sure that I know which calling convention are the functions in the dll using.
So (finally:) here's my question: can I deduce the calling convention programmaticaly? Dependency Walker won't help me, and I've no idea how to manually read PE format.
The answer is maybe.
If the functions names are C++ decorated, then you can determine the argument count and types from the name decoration, this is your best case scenario, and fairly likely if MSVC was used to write the code in the first place.
If the exported functions are stdcall calling convention (the default for windows api), you can determine the number of bytes to be pushed, but not the types of the arguments.
The bad news is that for C calling convention, there isn't any way to tell by looking at the symbol names. You would need to have access to the source code or the debug info.
http://en.wikipedia.org/wiki/X86_calling_conventions
The name that a function is given as an export is not required to have any relationship with the name that the linker sees, but most of the time, the exported name and the symbol name that the linker sees are the same.