Weird memory corruption

Question

This error only is occuring when I build with the WinDDK nmake compiler in 'free' (release), which performs optimizations. I can't reproduce this in the 'checked' build or compiling with VS.

Here is pseudo-code of what is happening in my code:

main()
{
   //variable init and other code
    fprintf(log, "pre nullValue: %lu\n", NULL);  //printf added for debugging
   otherFunc();
   funcWithError(str1, str2, NULL);
    fprintf(log, "post nullValue: %lu\n", NULL); 
    fprintf(log, "post2 nullValue: %lu, %lu, %lu\n", NULL, 0, NULL);
}

BOOL otherFunc()
{
   //variable init and other code
   callToDll();
    //...doing stuff
   libusb_usb_open(var1);    //If I remove this line there is no problem!!
    //...doing more stuff
}

BOOL funcWithError(char* s1, char* s2, FUNC_PTR fp)
{
    fprintf(log, "inFunc nullValue: %lu, %lu\n", NULL, fp);
   if(fp != NULL)
      return FALSE;        //This line is being executed erroneously!!
}

Output in log:
pre nullValue: 0
inFunc nullValue: 0, 251208
post nullValue: 251208
post2 nullValue: 251208, 251208, 251208
Note: The re-occurring number (251208) is a different number each time the program is run

Just changing that one line fixes/causes it. It is the libusb usb_open call.

1. Ultimately my question is to figure out how to fix the problem (I can't avoid that call)
2. But just on a stack/memory management level how is it even possible to have NULL not zero and have a literal value '0' print as non zero?

let me know what other information might help...

Answer 1

Not a completely slamdunk. But it is very likely that the stack is getting imbalanced. In the debugger (yes, you can debug the release build), check the value of the ESP register before and after the call. It should be the same. It won't be if the calling convention for the function is wrong. Like __stdcall vs __cdecl.

This can hide itself well when you build programs with nmake.exe, it is easy to forget to turn on the /RTCs option in the debug build so stack checking code is emitted. And the ESP register tends to restore itself on a function return. Until you build the release version where functions get inlined and usage of EBP is optimized away so ESP doesn't restore itself anymore.