I want to create thread or process that would have its own virtual address space (It would probably would have to be separate process) without system libraries in the address space. My goal is to create an execution environment for foreign origin code.
I would like to create a thread with no system libraries, just few executable pages where user's code would be copied and the thread entry point would be placed and also few RW pages for stack and data exchange with main thread.
Is it possible to completely unmap all system libraries on windows (or possibly Linux) from virtual memory from application level?
Unmapping system libraries will not prevent the binary from performing system calls by itself. To catch all operations which you are trying to prevent, some form of binary translation is necessary. You might want to have a look at libdetox and fastBT (Google Tech Talk about fastBT)