Search code examples
c++loadlibraryopenh264

Dynamically calling Openh264 functions causes my app to crash

Basically I'm trying to use the pre-built Openh264 DLLs from their webpage.

Using LoadLibrary and GetProcaddress works, until I call the required WelsCreateEncoder().

The VS error tells me that ESP was not saved correctly.

In the debugger I can just keep running and everything WORKS fine. I have been debugging it fine and I've got everything to work. but as a standalone app it just crashes.

I am assuming because it's a C library and my app is c++ that my calling conventions are wrong?

https://github.com/cisco/openh264

I added the ASM for the beginning and end of WelsCreateDecoder. is it me or does it look like it forgets to sub esp, 4 ?

typedef int(WINAPI * _pWelsCreateSVCEncoder)(ISVCEncoder** ppEncoder);
typedef void(WINAPI * _pWelsDestroySVCEncoder)(ISVCEncoder* ppEncoder);

0F3B8BF0 | 55                       | push ebp          |
0F3B8BF1 | 8BEC                     | mov ebp,esp       |

...

0F3B8C0F | 33C0                     | xor eax,eax       |
0F3B8C11 | 85C9                     | test ecx,ecx      |
0F3B8C13 | 0F94C0                   | sete al           |
0F3B8C16 | 5D                       | pop ebp           |
0F3B8C17 | C3                       | ret               |
Solution 
  • typedef int(* _pWelsCreateSVCEncoder)(ISVCEncoder** ppEncoder);
typedef void(* _pWelsDestroySVCEncoder)(ISVCEncoder* ppEncoder);

    turns out it was a lack of calling convention. no WINAPI needed.

    although this line https://github.com/cisco/openh264/blob/8533dd9daaaf3d7a51d9295686c31718ec3c946e/codec/decoder/plus/src/welsDecoderExt.cpp#L1052 confused me.