Custom calling convention for P/Invoke and C#

Question

I have a business case whereby I need to be able to specify my own calling convention when using P/Invoke. Specifically, I have a legacy dll which uses a non-standard ABI, and I need to able to specify the calling convention for each function.

For example, one function in this dll accepts its first two arguments via EAX and EBX, with the rest via stack. Another function accepts one argument via ECX, with the rest on the stack. I have a few hundred of these functions, and would like to avoid writing my own intermediate bridge DLL in order to access these functions.

My other option would be to hand-roll my own custom P/Invoke, which is undesirable for obvious reasons.

Any help is appreciated, thanks,

Answer 1

I don't understand what you mean with custom P/Invoke, but I can't see how you could get away without non-managed C++ with inline assembly. However, since almost everything is passed as 32-bit values, you might get away with writing only one proxy for each function signature, as apposed to one per function. Or you could write a code generator which generates proxies from XML. I can't see this version being too undesirable though, since all proxy functions will be really simple:

int RealFunction(int param1, const char * param2, char param 3);

int MyFunction(int param1, int param2, int param3) { // argument types do not matter as long as they are not doubles or structures
   __asm {
      mov eax, param1
      mov ebx, param2
      push param3
      call RealFunction
      ; depending on calling convention, you might need to do add esp, 12 here
      ; if RealFunction does not return its result in eax, you will need to do mov eax, <wherever the return value is> here
   }
}