Sandbox Virtual Machine for an Application (concept)

Question

I'd like to write a sandbox virtual machine for executing a compiled program. My goal is to isolate that program from the rest of operating system and control its execution so that it can't do anything harmful to a host computer.

I assume that:

• executed program is compiled to Portable Executable format and it's in machine code, not in any kind of byte code or for CLR,
• executed program is not allowed to communicate with peripherals like printer, scanner, and doesn't use any GUI,
• executed program's main task is to process some data stored in a local file (eg. calculations), and put its results in another local file,
• executed program shouldn't be able to communicate directly with an operating system, every request should be handled by a virtual machine, any request that may cause damage to an operating system should be blocked.

My concept of sandbox virtual machine's architecture and operation:

• application consists of several objects that simulate: processor, memory, i/o operations on files,
• there is a module that reads compiled file and loads executable code to a virtual memory,
• then the virtual processor starts processing from the first byte, reads opcode, arguments, loads them from memory if needed, executes command and puts the result in appropriate place, sets virtual flags if needed, then reads the next command, until the program is executed to the end.

What do you think: is it a good concept? What would you change to improve it?

Answer 1

Simulating a complete machine seems like a very slow way to execute native code. Lots of operations with load, lookup, execute, store, etc just for a single native instruction.

I would try to execute at least some blocks of code natively. Think of the following code.

int sum = 0;
for (int i = 0; i < 10; i++)
{
    sum += i;
}

This code is completely safe to execute native in your virtual machine. Just make sure that you inject a return call to your virtual machine code.

But I would try to go a step further and execute all code natively except for library/os calls. Before loading the sandboxed application, scan through the file and replace all "dangerous" calls with calls to handlers in your virtual machine.
The code

printf("Hello World\n");

would be replaced with calls to your library

myVM_printf("Hello World\n");

Then you can execute the whole program at native speed and still be able to handle all the dangerous code in your virtual machine.