I have a DLL I inject into other processes using SetWindowsHookEx. Inside the DLL I increment the module's reference counter by calling GetModuleHandleEx so I can control when the module is unloaded.
At this point the module reference count "should be" 2 from both of those API calls. When the calling process shuts down, it calls UnhookWindowsHookEx, decrementing the reference count to 1. The DLL has a thread that waits on a few things, one of them being the handle of the process that called SetWindowsHookEx. When the process goes away the DLL does some cleanup, terminates all threads, cleans up memory and handles and then calls FreeLibraryAndExitThread. This decrements the counter and the DLL gets unloaded.
Here's my problem. There are a few processes, especially those without a UI, where the DLL never gets unloaded. I'm pretty confident I have cleaned up everything. And I know for a fact that none of my threads are running.
First of all, if you have any troubleshooting tips to help uncover the cause, that would be helpful. Otherwise, I was thinking about using some API like NtQueryInformationProcess to get the module address and confirm the module handle count is in fact zero, then call CreateRemoteThread to inject a call to LdrUnloadDll to unload the module address from within the process. What are your thoughts to this approach? Does anyone have any example code? I'm having difficulty finding out how to get the module handle count.
I found the cause of the problem and the solution. I honestly feel quite stupid for missing it and struggling with it for so long.
As I mentioned in the original problem, the processes that are problematic do not have a UI. Turns out they do have a message pump running. The problem is nothing is sending messages to these processes without a UI after we call UnhookWindowsHookEx that would trigger the unloading. (In fact, I believe MSDN does state that window messages are not sent to processes when calling UnhookWindowsHookEx.)
By broadcasting WM_NULL to all processes after the injecting process calls UnhookWindowsHookEx the message pump wakes up in the injected processes and the DLL reference count is decremented. The DLL is unloaded immediately when the injected DLL finally calls FreeLibraryAndExitThread.
This is only part of the solution. If the injecting process is killed or crashes, no message is broadcasted so the DLL does not get unloaded from processes that do not have a UI. As I mention before I have a thread running in the DLL that waits on the injecting process handle. When the injecting process ends the DLL is signaled and then calls PostThreadMessage to send WM_NULL to each thread in the process. Then it waits until the DLL reference count is decremented before continuing and cleaning up before calling FreeLibraryAndExitThread. As a result, the DLL is unloaded almost immediately from all processes, UI or no UI.