Search code examples
c++linuxcalling-conventionabi

Calling-convention for the 'this' parameter on Linux x64

I have a scenario where I need to call a function from LLDB (but it could be any C++ API) on Linux x64, from an app written in a different language. For that reason, I need to properly understand the calling-convention and how the arguments are passed.

I am trying to call SBDebugger::GetCommandInterpreter, defined as:

  lldb::SBCommandInterpreter GetCommandInterpreter();

The full header file can be found here: https://github.com/llvm-mirror/lldb/blob/master/include/lldb/API/SBDebugger.h

My assumption was that the method would expect a pointer to SBDebugger as the this argument, in the RDI register. However, when calling the function that way, I get a segmentation fault.

If I look at the disassembly of the function, I see:

push   %r13 
push   %r12 
mov    %rsi,%r12 
push   %rbp 
push   %rbx 
mov    %rdi,%rbx

The function is reading from both RDI and RSI, despite only expecting the this argument. The only explanation I see is that the function is expecting this as a value rather than a reference. SBDebugger has a size of 16 bytes (one shared_ptr), and the calling convention states that a single 16 bytes parameter would be split into the RDI and RSI registers, which matches what I'm seeing.

However, that doesn't make sense to me for multiple reasons:

  • If this is passed by value, how would it work if the method has any side-effect? The changes wouldn't be reflected on the caller
  • The System V ABI states that: If a C++ object has either a non-trivial copy constructor or a non-trivial destructor 11, it is passed by invisible reference. The SBDebugger does have a custom destructor and copy-constructor:
  SBDebugger();

  SBDebugger(const lldb::SBDebugger &rhs);

  SBDebugger(const lldb::DebuggerSP &debugger_sp);

  ~SBDebugger();

Despite this, I tried calling the method by passing SBDebugger by value and it seems to work, but I get a segmentation fault later when I try to use the returned SBCommandInterpreter, so it's possible the method is only returning garbage.

There is something I don't understand about this method call, but I haven't been able to figure out what. What value should I pass in what registers, and why?

Solution

  • I figured it out. I was focusing on the arguments but the trick was the return value. Quoting the System V calling convention:

    A struct, class or union object can be returned from a function in registers only if it is sufficiently small and not too complex. If the object is too complex or doesn't fit into the appropriate registers then the caller must supply storage space for the object and pass a pointer to this space as a parameter to the function.

    Basically, because SBCommandInterpreter is considered as a complex object, the GetCommandInterpreter expects an additional hidden parameter, which is the address to which the return value will be written. So the "real" method signature is:

     lldb::SBCommandInterpreter* GetCommandInterpreter(SBCommandInterpreter& returnValue, SBDebugger* this);

    So the this argument is passed by reference as expected, I was just missing an additional hidden argument.