I'm reverse engineering an old game trying to get its compiled shaders bytecode so I can decompile it, I have hooked DrawIndexed and im grabbing the current pixel shader with PSGetShader. now all I have is the ID3D11PixelShader object. how can I get the shaders bytecode from just this object? I have looked through the ID3D11PixelShaders functions and there doesnt seem one to do this.
Hopefully some d3d wizard will know how to do this! thank you
(this seems to be easy in d3d9, IDirect3DPixelShader9 has a function called GetFunction that return the bytecode. there should be a d3d11 equivalent right?)
I have hooked DrawIndexed and im grabbing the current pixel shader with PSGetShader.
It seems you’ve already managed to hook D3D API calls with MinHook or similar?
Hook one more method, ID3D11Device.CreatePixelShader. Call the original implementation. Check if the result code is successful, also verify the output pointer received a newly created ID3D11PixelShader object. If both are true, you have a new shader, while the bytecode of the shader is still available.
I don’t know what do you want to do next. But note these shaders, and many other D3D objects, inherit from ID3D11DeviceChild COM interface. That interface has SetPrivateData and GetPrivateData methods which implement a dictionary with GUID keys and arbitrary values.
You can compute hash of the byte code as you were planning, save the shader binary so you can decompile it offline, but also store the hash of the shader in the newly created object, by calling SetPrivateData. Ideally, use guidgen.exe (the tool is included in visual studio, “Tools / Create GUID” menu item) to generate a new GUID for the dictionary key you use. Then, in the intercepted PSGetShader call you can call GetPrivateData() on the shader with that key, and you’ll get the hash code you previously stored in that shader, while you still had the byte code of the shader.