I got a very special problem. For a VM I need to copy code from the instruction functions to a ubyte array and then execute this array (the technic is similiar to the inline macro vm in gcc), basically it works like this:
__gshared void * sp = null, sb = null; //stack pointer and stack base
__gshared void add() //the function is just there to access the instruction code
{
asm{db "INSTRUCTIONCODESTART";} //this is a key to know where the instruction code starts
//instruction code here (sample instruction add, pops 2 values from the stack and pushes its result)
sp += 4;
*cast(uint*)sp += *cast(uint*)(sp - 4);
asm{db "INSTRUCTIONCODEEND";} //this is a key to know where instruction code ends
}
In the Init method, every instruction code gets its own buffer, and in the buffer is every byte between the INSTRUCTIONCODESTART and the INSTRUCTIONCODEEND key. I make this array executeable through the windows VirtualProtect call.
So Far, everything works as expected, but when I try to do a function call as an instruction, I will get an error.
__gshared void testcall(){}
__gshared void call()
{
asm{db "INSTRUCTIONCODESTART";} //this is a key to know where the instruction code starts
//instruction code here (just calls a D function)
testcall(); //this somehow throws an error
asm{db "INSTRUCTIONCODEEND";} //this is a key to know where instruction code ends
}
Btw I tested the instructions with the following code
void instructiontest()
{
uint dummy;
ubyte[] buf = getFunctionCode(&add) ~ 0xC3; //gets code of instruction, appends 0xC3 at it ("ret" instruction, for test purposes only to see if it returns to the D code without errors)
VirtualProtect(cast(void*)buf, buf.length, PAGE_EXECUTE_READWRITE, &dummy); //makes it executeable
dummy = cast(uint)&buf[0];
asm
{
call dummy[EBP];
}
print("instruction worked without errors!");
}
So far, every simple instruction (add, mul, sub, push0, push1, ...) works, but if I try to get the code of an instruction with a function call, it throws an error
I would be happy and very thankful about any help. (btw I need function calls in the instruction in order to let the script language communicate with D)
You should really disassemble the code in order to get a clear view of what it's doing and why your code is breaking. The disassembly for your call function is:
0000000000414db8 <_D4test4callFZv>:
414db8: 55 push rbp
414db9: 48 8b ec mov rbp,rsp
414dbc: 48 83 ec 08 sub rsp,0x8
414dc0: 53 push rbx
414dc1: 41 54 push r12
414dc3: 41 55 push r13
414dc5: 41 56 push r14
414dc7: 41 57 push r15
414dc9: 49 rex.WB
414dca: 4e 53 rex.WRX push rbx
414dcc: 54 push rsp
414dcd: 52 push rdx
414dce: 55 push rbp
414dcf: 43 54 rex.XB push r12
414dd1: 49 rex.WB
414dd2: 4f rex.WRXB
414dd3: 4e rex.WRX
414dd4: 43 rex.XB
414dd5: 4f rex.WRXB
414dd6: 44 rex.R
414dd7: 45 53 rex.RB push r11
414dd9: 54 push rsp
414dda: 41 52 push r10
414ddc: 54 push rsp
414ddd: e8 ce ff ff ff call 414db0 <_D4test8testcallFZv>
414de2: 49 rex.WB
414de3: 4e 53 rex.WRX push rbx
414de5: 54 push rsp
414de6: 52 push rdx
414de7: 55 push rbp
414de8: 43 54 rex.XB push r12
414dea: 49 rex.WB
414deb: 4f rex.WRXB
414dec: 4e rex.WRX
414ded: 43 rex.XB
414dee: 4f rex.WRXB
414def: 44 rex.R
414df0: 45 rex.RB
414df1: 45 rex.RB
414df2: 4e rex.WRX
414df3: 44 rex.R
414df4: 41 5f pop r15
414df6: 41 5e pop r14
414df8: 41 5d pop r13
414dfa: 41 5c pop r12
414dfc: 5b pop rbx
414dfd: c9 leave
414dfe: c3 ret
414dff: 90 nop
414dc9 is where the start marker begins, 414ddc is where it ends (inclusive). 414de2 is where your end marker begins, 414df3 is where it ends (inclusive). So, ripping that out, we have:
0000000000414db8 <_D4test4callFZv>:
414db8: 55 push rbp
414db9: 48 8b ec mov rbp,rsp
414dbc: 48 83 ec 08 sub rsp,0x8
414dc0: 53 push rbx
414dc1: 41 54 push r12
414dc3: 41 55 push r13
414dc5: 41 56 push r14
414dc7: 41 57 push r15
; code start marker here
414ddd: e8 ce ff ff ff call 414db0 <_D4test8testcallFZv>
; code end marker here
414df4: 41 5f pop r15
414df6: 41 5e pop r14
414df8: 41 5d pop r13
414dfa: 41 5c pop r12
414dfc: 5b pop rbx
414dfd: c9 leave
414dfe: c3 ret
414dff: 90 nop
You're clearly not copying some prologue and epilogue code here. But that, in and of itself, should not be terribly problematic.
I tried this program:
void main()
{
foo();
}
void foo()
{
auto addr = &bar;
asm { call addr; }
}
void bar()
{
asm { naked; call baz; ret; }
}
void baz()
{
}
It works for me. Frankly, I can't tell where your problem is. Most of the code you pasted can't just be copied into a source file and be compiled, so it's quite hard to tell what's going wrong. I hope some of the information here can help you. You're most likely going to have to attach a debugger and find out what's wrong; don't expect to mess with low-level stuff like this without getting your hands dirty. ;)
BTW, I tested on 64-bit x86 in Linux.
Either way, what you're doing is highly unportable, undefined, etc. Be warned. It may work out, but you're working with zero guarantees.