I am working on a game and thought about approaches for making it moddable. Usually, engines employ a scripting language for various reasons. One of them is that the engine gets to control what the mod sees, and can provide some degree of security for the user, by e.g. not allowing file access or network access, or constraining it in some way.
For my game I wanted to dig a bit into the possibility of providing mods as compiled DLLs. The problems here from a security perspective is that the dll can of course contain any code whatsoever, so I have absolutely no control over files/network etc. The dll code could also corrupt the game's memory. One approach to solve this would be to create a second process that simply has no rights to access the disk/network and can only use an API provided by the main game process. Memory could be shared between the processes, but before switching to the mods process, the pages could be marked as read-only, so there is no opportunity to corrupt the memory.
Is it possible to create such a process, that has basically no access to the system, or a controllable subset? I'm thinking about creating a whitelist for the windows API to e.g. allow allocations but disallow file access. All documentation about security I could find were about what OTHERS can do to/with the process, not what the process itself is allowed to do.
Using a restricted child process is definitively the right approach but there is not a single API you can call to just make this work.
You should probably start with CreateRestrictedToken and restrict the token as much as possible. If you are designing this from scratch you might be able to use SaferComputeTokenFromLevel to create a SAFER_LEVELID_UNTRUSTED or lower token but not everything works at this level.
Set the UAC integrity level to Low (or lower) (TokenIntegrityLevel). Ideally you would use an AppContainer but the documentation in this area of the API is not great. Google Chrome probably does it so you can look at how they do it.
You should now create a job object and at a minimum, set limits with JobObjectBasicUIRestrictions.
It is now time to start the process suspended with CreateProcessAsUser.
Add the process to the job and resume its execution.
The first thing the new process should do is to call SetProcessMitigationPolicy.
How you communicate with the child is a subject of its own. Perhaps CreateFileMapping to create a page file backed memory region. Use pipes or events to control the communication.