Search code examples
c++windowswinapiout-of-memoryshellexecute

What could possibly cause ShellExecute to return SE_ERR_OOM (error 8)?


I frequently call ShellExecute from a native C++/Win32 application to execute any shell item selected by the end-user from the GUI. Items are either executable files/scripts or links (.lnk). In some conditions that remain obscure to me, the following function sometimes returns 8 (SE_ERR_OOM error; only very briefly documented). As a result, the item is not executed. What could possibly cause this error?

int doExecute(LPCTSTR file, LPCTSTR args, LPCTSTR workDir)
{
    assert(file && *file);
    HRESULT hRes = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE);
    assert(hRes == S_OK || hRes == S_FALSE);
    int code = (int)ShellExecute(NULL, NULL, file, args, workDir, SW_SHOWNORMAL);
    doLog("ShellExecute returned: %d, %u", code, GetLastError()); // EDIT
    CoUninitialize();
    return code;
}

The context:

  • Windows8 64bit, application is 32bit, could not try on any other machine
  • Application is multi-threaded
  • By design, a new thread is always created specifically to do this call
  • The assertions in this code never break during my debug sessions
  • The application itself does not make direct use of COM but may have to call Win32 APIs that indirectly use it since it interacts quite frequently with the shell. In these cases, COM is always initialized with the same flags than shown above.
  • The flags passed to CoInitializeEx have been blindly chosen after MSDN recommendation (see ShellExecute documentation), not because of a personal choice

Observations about the bug so far:

  • It's rare
  • It occurred only after one or several hibernations (my application is always running on my laptop and I use it quite often)
  • An immediate call to GetLastError after calling ShellExecute always returned 0
  • As far as I can remember, it always happened while trying to execute a .lnk file, but not always the same file
  • A quick look at Process Explorer (sysinternals.com) running on the machine does not show any peak of use of memory
  • edit: I made a last test before posting here by calling doExecute 200 times in a row. All processes were spawned without error.

Solution

  • After quite a long run of tests, the bug never occurred anymore after I applied some changes to the code, following advises from @DavidHeffernan and @RossRidge. While I can't really consider this as a definitive answer per se since I still don't know what happened exactly under the hood, I haven't been able to reproduce the bug so far.

    Modifications applied:

    • Replaced the ShellExecute call by ShellExecuteEx.
    • Call CoInitializeEx once for all for every new thread, without its CoUninitialize counter-part. But kept the following nested CoInitializeEx-CoUninitialize pairs.

    Edit: In case someone needs it, just to confirm the problem didn't occur anymore, even after several months of tests, having these modifications applied.