I am spawning an application from c++ with ShellExecute, so I have the HINSTANCE of the app.
How can I close it now using that HINSTANCE? And can I use WaitForSingleObject() to wait for the app to finish?
First of all, an HINSTANCE is of very little use in modern versions of Windows -- but what you have isn't really an HINSTANCE anyway. The return from ShellExecute is really just a value greater than or less than 32, to indicate success or failure respectively.
Fortunately, if you use ShellExecuteEx, you can get a process handle for the new process, and you can use that to control the process.
The MSDN article that @Remus linked is decent, but (IMO) there's another step that can be useful if the target application is (or might be) a console application. In this case, it usually won't handle a WM_CLOSE message. You can, however, inject a DLL into the process, and have that do a clean(ish) shutdown from inside the process (for example, if it calls exit, and the target program is written in C, it'll get a chance to flush and close files, run anything registered with atexit, etc., before dying).
If that fails, you might want to use GenerateConsoleCtrlEvent to send it a control-break.
Then, if all those fail to exit you call TerminateProcess.