I want to create a Windows mutex by using WinAPI, CreateMutex() and OpenMutex(). But for security concern, I want the mutex be opened by those processes who know the "password" or the hard-code magic code. I don't want the mutex be accessed by every processes.
For example, Create mutex with name "Globel\cd689f00-0462-11e5-b939-0800200c9a66". So only the process who know the mutex name can access this mutex. But this is not a good solution because you can simply use Winobj.exe and you still have some chance to find this mutex. I want this mutex be protected by something like ACL (Access Control Lists). The problem is, I can't find a way to create my own SID for ACL.
Here is what I know and what I want: 1. I know I can make mutex be accessed by many process by naming it likes "Global\MyMutexName". 2. I also try to understand ACL and SID mentioned on MSDN. But I still can't find a way to create my own SID (maybe that doesn't make sense?). 3. I don't want to elevated my processes to Admin.
What you're trying to do isn't natural to the security model that Windows uses. Permissions are always granted based on who is running the executable, not on the executable itself. However, depending on your scenario, there may be suitable options.
If all of the processes involved will be in the context of the same user, then you can use IPC (e.g., named pipes) to identify your "friendly" processes and DuplicateHandle() to pass a handle to an unnamed mutex between processes.
If the processes must be in different user contexts, one option would be to have a system service running in a privileged context to act as a broker. Of course, that requires that the system service be installed with admin privilege, but this is usually acceptable; it only has to be done once.
If the application must be portable (no installation, or able to be installed without admin access) and must cross user boundaries then I think you would have to use IPC to implement your own mutex. Of course, that would be very inefficient.
Keep in mind that in any of these scenarios, a malicious user can still get access to the mutex - the best you can do is make it slightly more difficult. If you use an actual mutex, for example, the attacker could enumerate the handles of one of the processes, identify the mutex, and duplicate the handle. (Or simply inject malicious code into one of the supposedly "friendly" processes to use the handle directly.) Even the IPC could be spied upon and duplicated.
You should seriously consider whether the marginal security benefit is worth the significant increase in complexity. IMO, it isn't likely to be.
(The proper solution, by the way, is for a system service to do all the work that requires access to the mutex; the application that the users run would typically be a thin client that does nothing but provide the GUI.)