Msvc Inline assembly inner function calls

Question

first i'd like to say i researched this matter but couldn't find anything relevant related to it.

I am writting a c++ console program on msvc 2013 on release mode 32 bit. I am using inline assmebly in one of the file and it works great except when my inline assembly function_1 calls my function_2, when that happens there are intructions which get added to function_2 and therefore the stack gets damaged and the program crashs. If i stop using calls but merely "lea ebx,[eip+5]", push ebx, jmp xxxxxx instead it works fine.

So in my case and to be a little more concrete function 2 is defined like this :

    void test()
{

_asm{
f_01758630:  // ; <= Procedure Start

    PUSH EBP
            MOV EBP, ESP
            PUSH ESI
            PUSH EDI
            mov edi, [ebp + 0x0C]
            XOR ESI, ESI
            SHR EDI, 0x2
            TEST EDI, EDI
            JLE f_0175866B
            PUSH EBX
            mov ebx, [ebp + 0x08]

    f_01758645:

        MOV EDX, DWORD PTR DS : [EBX + ESI * 0x4]
            ROL EDX, 0x10
            MOV ECX, EDX
            MOV EAX, EDX
            SHR ECX, 0x8
            SHL EAX, 0x8
            XOR ECX, EAX
            SHL EDX, 0x8
            AND ECX, 0xFF00FF
            XOR ECX, EDX
            MOV DWORD PTR DS : [EBX + ESI * 0x4], ECX
            INC ESI
            CMP ESI, EDI
            JL TERA_01758645
            POP EBX

        f_0175866B :

        POP EDI
            POP ESI
            POP EBP
            RETN//; <= Procedure End
}
}

However when i debug the running program I can see the function is implemented like this :

push ebx push esi push edi push ebp mov ebp,esp push esi push edi

Ie msvc implemented 3 pushs, is that maybe related to _asm{} which is within the function any insight to how I can fix this ?

Answer 1

The first

push    ebp
mov     ebp, esp
push    ebx
push    esi
push    edi

is the automatically generated prologue of the function. At the end of the function is the epilogue:

pop     ebx
pop     edi
pop     esi
pop     ebx
pop     ebp
ret

Your _asm block has its own prologue and epilogue, so the code is done twice. Worse, the ret inside the _asm block gets a wrong return address and the program will crash. You can avoid the function prologue/epilogue by declaring the function as naked:

__declspec (naked) void test()
{
    _asm
    {
        own prolog
        ...
        own epilog
        ret
    }
}

This is dangerous, since you can forget to preserve registers that are to be returned unchanged, "callee saved registers": EBX, EBP, EDI, ESI. In MSVC inline assembly, it's easy to use the function arguments and the local variables, so it isn't necessary to take the control over epilog and prolog.

Look at this example (as close as possible to your code):

#include <stdio.h>

void function_2(unsigned* reg_ebx, unsigned reg_edi)
{
    _asm
    {
            mov edi, reg_edi      // take the second argument
            xor esi, esi
            shr edi, 2
            test edi, edi
            jle f_0175866b
            mov ebx, reg_ebx     // take the first argument

        f_01758645:

            mov edx, dword ptr ds : [ebx + esi * 0x4]
            rol edx, 16
            mov ecx, edx
            mov eax, edx
            shr ecx, 8
            shl eax, 8
            xor ecx, eax
            shl edx, 8
            and ecx, 0xff00ff
            xor ecx, edx
            mov dword ptr ds : [ebx + esi * 0x4], ecx
            inc esi
            cmp esi, edi
            jl f_01758645

        f_0175866b :
    }
}

void function_1 ()
{
    unsigned arr[8] = {1000,2000,3000,4000,5000,6000,7000,8000};
    int i;
    for (i=0; i < sizeof(arr)/sizeof(arr[0]); ++i) printf ("%08X ",arr[i]); puts ("");

    _asm
    {
        push LENGTH arr
        lea eax, arr
        push eax
        call function_2
    }

    for (i=0; i < sizeof(arr)/sizeof(arr[0]); ++i) printf ("%08X ",arr[i]); puts ("");
}

int main ( void )
{
    function_1();
    return 0;
}