Using SetWindowLong to override WndProc

Question

I need to set a WindowLong to override WndProc, I'm using this methods from coredll.dll:

public interface CoreDll extends StdCallLibrary {
	//loads the coredll with unicode options
    CoreDll INSTANCE = (CoreDll)Native.loadLibrary("coredll", CoreDll.class,
                                                   W32APIOptions.UNICODE_OPTIONS);
    //native calls
    HMODULE GetModuleHandle(String lpModuleName);
    long GetWindowLong(HWND hwnd, int gwlWndproc);
    long SetWindowLong(HWND hWnd,int nIndex,Callback dwNewLong);
    LRESULT DefWindowProc(HWND hWnd, int uMsg, WPARAM uParam,LPARAM lParam);
  }

The original method signature in coredll is:

LONG SetWindowLong( 
  HWND hWnd, 
  int nIndex, 
  LONG dwNewLong
); 

But when I'm using it it always return 0, according to Microsoft "The previous value of the specified 32-bit integer indicates success. Zero indicates failure". Anyway, I'm trying to execute it this way:

public interface CallbackProc extends Callback, StdCall {
	LRESULT callback(HWND hWnd, int uMsg, WPARAM uParam, LPARAM lParam);
}

//Get a handle to the current process
final HWND mainHwnd = CoreDll.INSTANCE.GetModuleHandle(null);

//Get a reference to the current process to send it in the new WndProc
long value=CoreDll.INSTANCE.GetWindowLong(new HWND(mainHwnd.getPointer()), -4);
final LONG_PTR prevWndProc = new LONG_PTR(value); 

//Sets the new Method to override WndProc
final RfidCallbackProc ptr=new RfidCallbackProc() {
	@Override
   	public LRESULT callback(HWND hWnd, int uMsg, WPARAM wParam,LPARAM lParam) {
        //returns the call to the process  
        return CoreDll.INSTANCE.CallWindowProc(prevWndProc, hWnd, uMsg, wParam, lParam);
    }
};

//Sets the new method to override Windows' WndProc
int num=(int)CoreDll.INSTANCE.SetWindowLong(new HWND(mainHwnd.getPointer()),-4 ,ptr);
//NUM IS ALWAYS 0

If someone can give me a hint i will really appreciate it, thanks.

Answer 1

You are retrieving the module handle (GetModuleHandle), this handle reference an executable file (exe or dll) loaded into a process space, not a window. SetWindowsLong fails because the object you passed to it is not a window. IIRC you have a method on forms/controls to get the native HWND, this is the right kind of object that you should pass to SetWindowLong. Looking at the code I suppose that you have a RFID library that sends codes via windows messages. You may consider to wrap this with some native code into an object that then generates "regular" .NET events. It would be a bit more complicated but avoids some of the headaches that SetWindowLong may generate (ex: if the object with the wndproc is destroyed when the window is still valid).