Under what conditions does RmGetList return 2 for the lpdwRebootReasons output parameter?

Question

Background

I am designing an Inno Setup installer to install a Cygwin service, and I am puzzled by the behavior I'm seeing from the Windows Restart Manager APIs.

Specifically, when the service is running (started using the cygrunsrv utility), the RmGetList API function returns 2 (RmRebootReasonSessionMismatch) for its lpdwRebootReasons output parameter. This output parameter is an enumeration of type RM_REBOOT_REASON, and the description on MSDN for the RmRebootReasonSessionMismatch value is:

One or more processes are running in another Terminal Services session.

The Inno Setup log file contains lines like the following:

RestartManager found an application using one of our files: <executable name>
RestartManager found an application using one of our files: <service name>
Can use RestartManager to avoid reboot? No (2: Session Mismatch)

Inno Setup then proceeds trying to replace in-use files, as if Restart Manager was not used at all.

I am puzzled by this output value, because on two different machines I tested (Windows 10 1909 x64 and Windows Server 2012 R2), no terminal server/remote desktop users are logged on.

If I stop the service and start another executable (in the set of files to be replaced by the installer), RmGetList returns 0 (RmRebootReasonNone) for lpdwRebootReasons, and Inno Setup displays the normal dialog for in-use files and allows the user to select to automatically close them.

Process Explorer shows both processes (cygrunsrv.exe and the process it starts) running in session 0 and at System integrity level. Both are console subsystem executables.

Questions

1. Under what conditions does RmGetList return 2 (RmRebootReasonSessionMismatch) for its lpdwRebootReasons output parameter? (I'm trying to understand why this happens when the service is running.)

2. Does this value cause the entire Restart Manager session to fail, or can Restart Manager proceed even though it thinks applications are running in one or more different sessions?

Answer 1

A Clue from the RestartManager PowerShell Module

The RestartManager PowerShell module (special thanks to Heath Stewart) provides a simple PowerShell interface for the Restart Manager. My commands are as follows:

Set-Location <path to Cygwin root directory>
Start-RestartManagerSession
Get-ChildItem . -File -Include *.exe,*.dll -Recurse | RegisterRestartManagerResource
Get-RestartManagerProcess
Stop-RestartManagerProcess

These commands produce the following output:

Id                : <process ID>
StartTime         : <process start time>
Description       : <executable started by cygrunsrv>
ServiceName       :
ApplicationType   : Console
ApplicationStatus : Running
IsRestartable     : False
RebootReason      : SessionMismatch

Id                : <cygrunsrv process id>
StartTime         : <cygrunsrv process start time>
Description       : <description of service>
ServiceName       : <service name>
ApplicationType   : Service
ApplicationStatus : Running
IsRestartable     : True
RebootReason      : SessionMismatch

For some reason, Restart Manager sees the cygrunsrv.exe service process as restartable, but the executable it spawns as not restartable. (I am still curious about why this happens in the first place.)

An Imperfect Workaround Attempt

Based on this observed behavior, I first attempted the following workaround:

1. In the Inno Setup script's [Setup] section, set the following:

    CloseApplications=yes
    CloseApplicationsFilter=*.chm,*.pdf
    RestartApplications=yes
   

   The CloseApplicationsFilter directive specifies what files get registered with the Restart Manager. Note I do not specify *.exe or *.dll here; I want to manually specify only certain .exe files in the [Code] section.

2. Call the Inno Setup RegisterExtraCloseApplicationsResource function once for each .exe file in the setup that will NOT be spawned by cygrunsrv and put them in the RegisterExtraCloseApplicationsResources event procedure. Example:

    [Code]
   
    procedure RegisterExtraCloseApplicationsResources();
    begin
      RegisterExtraCloseApplicationsResource(false, ExpandConstant('{app}\bin\cygrunsrv.exe'));
    end;
   

It's important not to register any executable spawned by cygrunsrv.exe or any of the Cygwin DLL files, because this will prevent the Restart Manager from taking effect in Inno Setup.

This solution is far from perfect, because executables normally started by cygrunsrv, if started separately, are not detected by Restart Manager (e.g., sshd.exe). For example, new SSH sessions are spawned in executables that the Restart Manager are not restartable.

A Better Solution

I've decided that a better solution is to detect any running executables from code and prompt the user apart from the Restart Manager functionality (which, simply put, doesn't work for Cygwin services).