I'm writing a debugger for my text editor in Windows X64. I implemented breakpoints and step-over/step-out. I'm stuck at step-in because I basically need to get the address of the instruction at the current line (which I get via the PDB file and DbgHelp) and the next line, see if there's a 'call' instruction between those addresses, if so we step-in else step-over.
I was wondering if there was a way to get this information (without having to write or use a disassembler) via DbgHelp/PDB? or maybe take advantage of certain byte order of instructions maybe the 'call' is unique and there's only one way its bytes are ordered so I can read byte by byte and see if they match the 'call'? (that way I don't have to do full or even partial disasm)
Ideas? Any help would be appreciated
Unfortunately, there is not.
The most common opcode for callq in x86_64 code is e8, but there's no way to distinguish an e8 meaning callq from an e8 in another context (e.g, as part of a constant) without fully disassembling the binary to figure out where each instruction starts. Remember that x86 uses variable-length instructions, so it's not always obvious where instructions start.