CoCreateInstance does not start or connect to ATL COM service

Question

I have a ATL COM service exe (MyService.exe), which compiles and runs fine. If I install this service (via MyService.exe /Service), it is successfully installed into the SCM. I can start the service through the SCM and it runs fine, under the LOCALSYSTEM account.

My problem arises when I attempt to create an instance of a COM class defined by the service. My test harness application (MyServiceTest.exe), calls the following:

::CoInitialize(NULL);
::CoInitializeSecurity(NULL, 
                        NULL, 
                        NULL, 
                        NULL, 
                        RPC_C_AUTHN_LEVEL_PKT, 
                        RPC_C_IMP_LEVEL_IMPERSONATE, 
                        NULL, 
                        EOAC_NONE, 
                        NULL);
ATL::CComPtr<IMyServiceInterface> pInterface;
HRESULT hr = CoCreateInstance(CLSID_MyServiceInterface, NULL, CLSCTX_LOCAL_SERVER, IID_IMyServiceInterface, reinterpret_cast<void**>(&pInterface));

At the call to CoCreateInstance, a few different things happen, depending on how MyService.exe is installed:

1. MyService.exe is installed using the /Service command line:
   MyServiceTest.exe calls CoCreateInstance, and MyService WinMain is called. CAtlServiceModuleT::Start is then called, which determines that the executable has been started with the command line option '-Embedding'. It determines that it's installed as a service, and as such calls ::StartServiceCtrlDispatcher(). This call fails with error code 1063 (ERROR_FAILED_SERVICE_CONTROLLER_CONNECT). According to MS:

"This error is returned if the program is being run as a console application rather than as a service. If the program will be run as a console application for debugging purposes, structure it such that service-specific code is not called when this error is returned."

The call fails, MyService.exe exits, and the CoCreateInstance call times out.

1. MyService.exe is NOT installed as a service, but is registered via /RegServer
   MyServiceTest.exe calls CoCreateInstance, and MyService WinMain is called. MyService.exe is instantiated under the logged in user account (not LOCALSYSTEM). The executable successfully runs, but not as a service, which is not the desired behaviour. Despite not running as a service, the CoCreateInstance() call succeeds, and I get a valid interface pointer through which I can call MyService COM functions.

2. MyService.exe is NOT installed as a service, is registered via /RegServer, and is already running (after successfully being started in scenario 2 for example)
   MyServiceTest.exe calls CoCreateInstance, and a NEW instance of MyService.exe is instantiated, again under the logged in user account. This behaviour continues for every subsequent call to CoCreateInstance.

My desired behaviour is that I can install MyService.exe as a service, and CoCreateInstance will start the server, or connect to the current MyService.exe instance if the service is already running. As far as I was aware, the above code should behave this way. What am I missing?

It seems the fact that the service runs under LOCALSYSTEM while the simple RegServer alternative runs under the local user could be pertinent, but I'm not sure if this is the issue.

The service side call to CoInitializeSecurity is:

HRESULT hr = CoInitializeSecurity(0,
                                    -1,
                                    0,
                                    0,
                                    RPC_C_AUTHN_LEVEL_PKT,
                                    RPC_C_IMP_LEVEL_IMPERSONATE,
                                    0,
                                    EOAC_NONE,
                                    0);

What am I doing wrong?

P.S. MyService.exe should not exit once started, as it contains a WaitForSingleObject() in the Run() function which is waiting on an external signal (which is also set in OnStop() so the SCM can halt the service). This is why MyService.exe persists after MyServiceTest.exe completes. This is desired behaviour (for a service, which is what it should be running as).

Answer 1

It turns out the culprit was how the service was registered. In order for a class to launch it's controlling application as a service, the controlling application needs to add entries into the registry so it is recognised as a local server, i.e:

(MyService.rgs)

HKCR
{
    NoRemove AppID
    {
        ForceRemove {6E5B1E7E-3340-4553-A356-76F1C3543452} = s 'MyService'
        {
            val LocalService = s 'MyService'
            val ServiceParameters = s '-Service'
        }

        'MyService.EXE'
        {
            val AppID = s {6E5B1E7E-3340-4553-A356-76F1C3543452}
        }
    }
}

The AppID of which is specified in MyService.rgs.

This leads to the following layout in the registry:

HKCR
    AppID
        {6E5B1E7E-3340-4553-A356-76F1C3543452} (Contains LocalService, ServiceParameters REG_SZ's)
        MyService.EXE (Contains AppID REG_SZ)
    CLSID
        {MyServiceInterface GUID} (Contains MyService.EXE AppID)

Pertinent links:
LocalService value
LocalServer32 overload in CoClass CLSID
Specifying AppID for CLSID