This is something I've been curious about for a while: I was wondering how LuaJIT's FFI module manages to use the correct calling conventions for invoking external native functions without any need for declarations in the user's prototypes.
I tried reading through the source code to figure this out on my own, but finding what I was looking for proved to be too difficult, so any help would be appreciated.
In order to verify that calling conventions are auto-determined when not declared, I wrote the following 32-bit test DLL to be compiled with MSVC's C compiler:
// Use multibyte characters for our default char type
#define _MBCS 1
// Speed up build process with minimal headers.
#define WIN32_LEAN_AND_MEAN
#define VC_EXTRALEAN
// System includes
#include <windows.h>
#include <stdio.h>
#define CALLCONV_TEST(CCONV) \
int __##CCONV test_##CCONV(int arg1, float arg2, const char* arg3) \
{ \
return CALLCONV_WORK(arg1, arg2, arg3); \
__pragma(comment(linker, "/EXPORT:" __FUNCTION__ "=" __FUNCDNAME__ )) \
}
#define CALLCONV_WORK(arg1,arg2,arg3) \
test_calls_work(__FUNCTION__, arg1, arg2, arg3, __COUNTER__);
static int test_calls_work(const char* funcname, int arg1, float arg2, const char* arg3, int retcode)
{
printf("[%s call]\n", funcname);
printf(" arg1 => %d\n", arg1);
printf(" arg2 => %f\n", arg2);
printf(" arg3 => \"%s\"\n", arg3);
printf(" <= return %d\n", retcode);
return retcode;
}
CALLCONV_TEST(cdecl) // => int __cdecl test_cdecl(int arg1, float arg2, const char* arg3);
CALLCONV_TEST(stdcall) // => int __stdcall test_stdcall(int arg1, float arg2, const char* arg3);
CALLCONV_TEST(fastcall) // => int __fastcall test_fastcall(int arg1, float arg2, const char* arg3);
BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID lpReserved)
{
if(dwReason == DLL_PROCESS_ATTACH) {
DisableThreadLibraryCalls(hInstance);
}
return TRUE;
}
I then wrote an LUA script for calling the exported functions with the ffi module:
local ffi = require('ffi')
local testdll = ffi.load('ljffi-test.dll')
ffi.cdef[[
int test_cdecl(int arg1, float arg2, const char* arg3);
int test_stdcall(int arg1, float arg2, const char* arg3);
int test_fastcall(int arg1, float arg2, const char* arg3);
]]
local function run_tests(arg1, arg2, arg3)
local function cconv_test(name)
local funcname = 'test_' .. name
local handler = testdll[funcname]
local ret = tonumber(handler(arg1, arg2, arg3))
print(string.format(' => got %d\n', ret))
end
cconv_test('cdecl')
cconv_test('stdcall')
cconv_test('fastcall')
end
run_tests(3, 1.33, 'string value')
After compiling the DLL and running the script, I received the following output:
[test_cdecl call]
arg1 => 3
arg2 => 1.330000
arg3 => "string value"
<= return 0
=> got 0
[test_stdcall call]
arg1 => 3
arg2 => 1.330000
arg3 => "string value"
<= return 1
=> got 1
[test_fastcall call]
arg1 => 0
arg2 => 0.000000
arg3 => "(null)"
<= return 2
=> got 2
As you can see, the ffi module accurately resolve the calling conventions for the __cdecl calling convention and the __stdcall calling convention. (but appears to have called the __fastcall function incorrectly)
Lastly, I've included dumpbin's output to show that all functions are being exported with undecorated names.
> dumpbin.exe /EXPORTS ljffi-test.dll
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file ljffi-test.dll
File Type: DLL
Section contains the following exports for ljffi-test.dll
00000000 characteristics
548838D4 time date stamp Wed Dec 10 04:13:08 2014
0.00 version
1 ordinal base
3 number of functions
3 number of names
ordinal hint RVA name
1 0 00001000 test_cdecl
2 1 000010C0 test_fastcall
3 2 00001060 test_stdcall
Summary
1000 .data
1000 .rdata
1000 .reloc
1000 .text
Just to clarify, since calling conventions are only really relevant for 32-bit Windows compilers, so that is the primary focus for this question. (Unless I'm mistaken, compilers targeting the Win64 platform only use the FASTCALL calling convention, and GCC uses the CDECL calling convention for all other platforms supported by LuaJIT)
As far as I know, the only place to find information about functions exported from a PE file is the IMAGE_EXPORT_DIRECTORY, and if function names are exported without decorators, there is no information remaining that indicates the calling convention of a particular function.
Following that logic, the only remaining method I can think of for determining a function's calling convention is to analyze the assembly of the exported function, and determine the convention based on the stack usage. That seems like a bit much, though, when I consider the differences produced by different compilers and optimization levels.
Calling convention is something platform dependent. Usually there is one platform's default and you may specify others.
From http://luajit.org/ext_ffi_semantics.html:
The C parser complies to the C99 language standard plus the following extensions:
...
GCC attribute with the following attributes: aligned, packed, mode, vector_size, cdecl, fastcall, stdcall, thiscall.
...
MSVC __cdecl, __fastcall, __stdcall, __thiscall, __ptr32, __ptr64,
Most interesting is Win32. Here calling convention maybe encoded with decorators Win32 calling conventions.
LuaJIT has code to recognize decorators.
Also, LuaJIT by default use __stdcall call convention for WinAPI Dlls: kernel32.dll, user32.dll and gdi32.dll.