Search code examples
cassemblygcccalling-convention68000

How to tell gcc to not align function parameters on the stack?

I am trying to decompile an executable for the 68000 processor into C code, replacing the original subroutines with C functions one by one.

The problem I faced is that I don't know how to make gcc use the calling convention that matches the one used in the original program. I need the parameters on the stack to be packed, not aligned.

Let's say we have the following function

int fun(char arg1, short arg2, int arg3) {
    return arg1 + arg2 + arg3;
}

If we compile it with

gcc -m68000 -Os -fomit-frame-pointer -S source.c

we get the following output

fun:
    move.b 7(%sp),%d0
    ext.w %d0
    move.w 10(%sp),%a0
    lea (%a0,%d0.w),%a0
    move.l %a0,%d0
    add.l 12(%sp),%d0
    rts

As we can see, the compiler assumed that parameters have addresses 7(%sp), 10(%sp) and 12(%sp):

illustration of the unwanted parameter positioning on the stack

but to work with the original program they need to have addresses 4(%sp), 5(%sp) and 7(%sp):

illustration of the desired parameter position

One possible solution is to write the function in the following way (the processor is big-endian):

int fun(int bytes4to7, int bytes8to11) {
    char arg1 = bytes4to7>>24;
    short arg2 = (bytes4to7>>8)&0xffff;
    int arg3 = ((bytes4to7&0xff)<<24) | (bytes8to11>>8);
    return arg1 + arg2 + arg3;
}

However, the code looks messy, and I was wondering: is there a way to both keep the code clean and achieve the desired result?

UPD: I made a mistake. The offsets I'm looking for are actually 5(%sp), 6(%sp) and 8(%sp) (the char-s should be aligned with the short-s, but the short-s and the int-s are still packed):

illustration of the updated desired parameter position

Hopefully, this doesn't change the essence of the question.

UPD 2: It turns out that the 68000 C Compiler by Sierra Systems gives the described offsets (as in UPD, with 2-byte alignment).

However, the question is about tweaking calling conventions in gcc (or perhaps another modern compiler).

Solution

  • To get integral parameters passed using 2 byte alignment instead of 4 byte alignment, you can change the default int size to be 16 bit by -mshort. You need to replace all int in your code by long (if you want them to be 32 bit wide). The crude way to do that is to also pass -Dint=long to your compiler. Obviously, you will break ABI compatibility to object files compiled with -mno-short (which appears to be the default for gcc).