I am following the process to create a trampoline in order to hook a dll function (in my case Direct3DCreate9 from d3d9.dll) as outlined here: https://www.malwaretech.com/2015/01/inline-hooking-for-programmers-part-1.html and https://www.malwaretech.com/2015/01/inline-hooking-for-programmers-part-2.html
My code differs slightly as I am working out the offset bytes manually using a disassembler rather than using the hde32_disasm function.
Everything seems to work fine, the victim process calls my injected dll wrapper function, the new function does some stuff, then calls the original function (Direct3DCreate9), once the original returns the wrapper should then call some other stuff, before returning to the victim process.
Unfortunately when the original function is called from the hook wrapper it returns back to the victim application and not the hook wrapper, this means it misses out some of the code from the wrapper.
Having stepped through the disassembly it looks as though the call stack is being overwritten so when the Direct3DCreate9 returns it pops back to the victim application instead of my hook function that made the call.
I'm guessing I need to push the hook function manually onto the call-stack? How would I go about this?
Other potentially pertinent information: Both the victim process and the hook have been built in debug mode. Direct3DCreate9 is a __stdcall and I am using vs2010 for the hook dll, but the victim process was compiled with vs2015.
It turns out the callstack was being knobbled by the NVidia graphics driver nvd3d9wrap.dll. This dll was injecting into the d3d9 application in the same manner as what I was trying to do. This lead to the madness explained in the original post.
The solution was to open Device Manager in windows and disable the NVidia graphics driver. Thankfully my pc has an integrated graphics chip, so I am able to use that.