Search code examples
c++windowscomactivex

Why does calling ShellExecute in Win Service cause unrelated incoming COM connections to fail with E_ACCESSDENIED?

The product I maintain has a Windows Service written in C++, and that service provides a COM interface that a hosted scripting environment (hosted by dllhost) can call into so that users can write scripts in powershell and have them interface with the service. The service maintains several COM classes which inherit from IActiveScriptSite/IActiveScript to provide this functionality to the hosted environment.

This functionality has been working well for a while now, but we recently had a bug where the whole thing broke. The bug manifested its self when the hosted environment started getting E_ACCESSDENIED when calling the OnStateChange() function on the IActiveScriptSite interface. This happened to be the first call to the interface.

After working out which change caused the problem, we eventually found that the whole issue can be reproduced by putting a single ShellExecute() call in the initialization of our service like so:

HRESULT CServiceObject::Run(int nShowCmd)
{
    ShellExecuteW(nullptr, nullptr, L"ipconfig", L"/all", nullptr, SW_HIDE);

    // Other initializeation code which ultimately 
    // leads to a thread which starts the hosted environment

This is seemingly in a completely unrelated area of the code to the code that's having the problem.

Reading the ShellExecute documentation lead me to try:

  • Initializing COM prior to the ShellExecute (this didn't make any difference whichever apartment model COM was initialized with, we normally use MTA throughout)
  • Trying to launch ShellExecute on it's own thread (this didn't make any difference)
  • Replacing ShellExecute with the ShellExecuteEx version and making it synchronous so that the process finished before anything else occured (this didn't make any difference)

So, bafflingly, ShellExecute is breaking our whole service. Can anyone think of what it might be doing which affects the state of the program after its run? I can actually fix this issue by just not calling ShellExecute() but it's worrying that something so trivial and seemingly unrelated to the other code can break it so I want to understand what the problem is.

Solution

  • This problem turned out to be because our code was calling CoInitializeSecurity(). Calling ShellExecute() before this results in CoInitializeSecurity() security being called implicitly and then it was subsequently failing when we called it which prevented our hosted scripting environment to contact the service.