Does Tomcat sandbox apps, like running them on a separate JVM or of some sort?
Tomcat doesn't really Sandbox web apps. The apps run in the same JVM. The only thing that might isolate an app is the classloader. Each webapp has its own classloader so it doesn't share app specific classes with other apps.
When you run out of memory, the container is out of memory also. The only reason it's running is probably because it doesn't need to allocate more memory.