Release-Build of DLL behaves unexpected

Question

I have a VS2013 project to build a DLL which I call from Java using JNA. Everything works expected, as long as I use the Debug Configuration - but when using any optimization (Configuration Properties -> C/C++ -> Optimization -> Everything else than "Disabled (/Od)"), the behavior becomes something unexpected. I want to understand why that is the case - do I'm relying on undefined-behavior in this case?

My expectation on calling createObject() twice is: first call should return true (because instance is uninitialized and will be initialized), second call should return false (because instance is supposed to be initialized already). However, if I use the Release-Build (or activate Optimizations in Debug-Build) createObject() is returning true for every consecutive call.

My C++ Code:

#include <windows.h> 
#include <memory>

#define DLL_EXPORT

#ifdef DLL_EXPORT
#    define CALLCONV extern "C" __declspec(dllexport)
#else
#    define CALLCONV extern "C" __declspec(dllimport)
#endif

class MyType{

};

std::unique_ptr<MyType> instance = nullptr;

BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason, LPVOID lpvReserved){
  switch (fdwReason){
  case DLL_PROCESS_ATTACH:
    break;
  case DLL_PROCESS_DETACH:
    break;
  case DLL_THREAD_ATTACH:
    break;
  case DLL_THREAD_DETACH:
    break;
  default:
    break;
  }
  return TRUE;
}

CALLCONV bool createObject(){
  bool retVal = true;
  if (instance == nullptr){
    instance = std::unique_ptr<MyType>(new MyType());
  }else{
    retVal = false;
  }
  return retVal;
}

Just for reference, my calling JNA code (but I guess, the same problem would be there, if I call it from native code as well):

import com.sun.jna.Library;
import com.sun.jna.Native;

public class main {
  public interface TestLibrary extends Library {
    TestLibrary INSTANCE = (TestLibrary)Native.loadLibrary("Test", TestLibrary.class);
    boolean createObject();
  }

  public static void main(String[] args) {
    try {
      System.out.println(TestLibrary.INSTANCE.createObject());
      System.out.println(TestLibrary.INSTANCE.createObject());
    } catch (UnsatisfiedLinkError e) {
      System.err.println("Native code library failed to load.\n" + e);
      System.exit(1);
    }
  }
}

Answer 1

The size of bool is implementation-dependent. JNA assumes a default conversion to boolean from a native int. You should ensure your native code is returning something with a well-defined size so that JNA can do the conversion reliably.

JNA requires well-defined sizes for everything in order to perform Java to native translation correctly.