I am using Google's v8 engine for embedding javascript in my application. At certain times, I will be invoking user-supplied code, and I want to ensure that it does not behave badly by allocating too much memory. Currently, when the javascript tries to make or resize an array to be too large, for example, I get an unceremonious message:
#
# Fatal error in CALL_AND_RETRY_LAST
# Allocation failed - process out of memory
#
And then entire process crashes with a SIGILL. Obviously, this is not acceptable. I require the ability to run user-supplied code, however... and it is not feasible to manually vet all code before it is executed by the engine.
What I would ideally like to do in this case is simply terminate the isolate that was consuming too much memory (without affecting any other isolates that may be running). Is there any way to designate a maximum amount of memory that a js program is allowed to use before it fails, and so if it exceeds that limit, instead of crashing the process, the invocation of the Run or Call commands would simply return an error or set some status flag indicating that it was abnormally terminated.
Things I have tried so far:
Setting a custom array_buffer allocator when the isolate is created, which tracks how much memory is being used and terminates the isolate when the memory usage gets too high> The Allocate function of my allocator does not ever get called.
Calling AddMemoryAllocationCallback with a function that tracks memory usage and tries to terminate the isolate via TerminateExecution() when the allocations exceeds a certain amount. This function does get called, but I get an out of memory error after this function only has reported a couple of megabytes being used, while I know for a fact that the data being created by the bad behaving v8 function is FAR larger than that.
Setting a fatal error handler via SetFatalErrorHandler and trying to invoke TerminateExecution there. This function does get called, but it does not prevent the process from crashing.
Is there anything else I can try?
Edit: authoritative response from the V8 team -- you can't. But they'll accept a patch.
v8::Isolate::SetFatalErrorHandler() should allow you to not crash. But, my understanding is that the isolate is still unusable after the fact. Probably no way of getting around that, since the isolate will be left in an unrecoverable statea.
http://v8.paulfryzel.com/docs/master/classv8_1_1_isolate.html#a131f1e2e6a80618ac3c8c266a041851d
(maybe. there seems to be a lot of stuff going on about this in the 2013-2014 timeframe where people at google said the right thing to do was to just let v8 kill the process -- to which a lot of people thought was dumb. I don't see any resolution)
edit: Mailing list response was that you cannot do this. They will accept a patch if it doesn't have a performance impact.
edit: There was just another thread on this and someone posted what seems to be a pretty good way to avoid OOM in non-malicious situations:
https://groups.google.com/forum/#!topic/v8-users/vKn1hVs8KNQ
I set the heap limit to 8x the limit I actually want. Then, after each call into the isolate, I check if the memory usage has gone over the intended limit. If so, I invoke garbage collection. If it's still over the limit after that, then I terminate the isolate at that point.
Meanwhile, we also enforce a CPU time limit of 50ms. In practice, a script that allocates tons of memory tends to run out of CPU time before it can hit the 8x heap limit (especially as the GC slows things down when approaching the limit).