I have a Win service running under the SYSTEM account. In case the user logs out from the system, the service should detect this and restart particular application on the logon desktop (and stop itself in case than user closing this application manually). The obvious way for me is detect which desktop (Default, ScreenSaver or Winlogon) is active now, but it seems that OpenInputDesktop call doesn't work under the service.
Services run in a different session than users do. Desktops (and other UI resources) belong to the Session they are created in, and cannot be accessed across session boundaries. So the service simply can't directly access a user's desktops at all.
To access a given user's desktop, you will have to run a separate process in that user's session, and that process can then communicate back to the service process as needed via any IPC mechanism of your choosing.
The service can monitor for SERVICE_CONTROL_SESSIONCHANGE events from the SCM to detect user logins/logouts, and spawn a process in a desired user session by using WTSQueryUserToken() and CreateProcessAsUser().