Defending against stack overflow by user scripts

Question

C++ has limited stack space but no way for functions to check whether there's enough space left for them to run. I don't know what to do about this when writing script bindings.

For example:

class Container : Widget {
  void addChild(WidgetPtr child) { ... }

  void draw(Canvas& canvas) {
    for (auto child : m_children) {
      child.draw();
    }
  }
};

A malicious script can do this to crash the program:

var a = new Container()
for (i = 0; i < 10000000; i++) {
  var b = new Container()
  a.addChild(b)
  a = b
}

a.draw() // 10000000 nested calls ---> stack overflow

There's also callback problem:

void doSomething(std::function<void()> callback) {
  callback();
}

If wrapped using something like this:

ScriptValue doSomething_Wrapper(ScriptArgs args) {
  doSomething([&]() { args[0].callAsFunction(); });
}

Crashed using:

function badCallback() { doSomething(badCallback) }
doSomething(badCallback)

...
doSomething_Wrapper
doSomething
ScriptValue::callAsFunction
...
doSomething_Wrapper
doSomething
ScriptValue::callAsFunction
...
BOOM!

What's the most idiomatic way to defend against this with least inconvenience?

What do browsers written in C++ (Firefox, Chrome) do?

What can I do not to introduce a vulnerability like this by accident?

Answer 1

While a "malicious" script could cause a stack overflow, like you describe, it can´t harm the program more than causing crashes that way (at least on modern OS where the stack limit is actually checked and therefore it´s safe against overwrites of other important data).

If it´s critical that the program is running all the time, another process has to monitor it (and restart it if necessary. Not only because stack overflows, there are many more potential problems.).
Other than that, there isn´t much one can do if the OS stack is used. Dynamically allocating a large memory block for a single pointer in the stack and doing the whole memory management manually in this block is possible, but maybe impractical.

About eg. FIrefox: At least parts of the program are using an own memory management (but I´m not sure if it is relevant for plugins, scripts etc.). Additionally, there is a separate process plugin-container.exe (at least on Windows), and killing won´t kill Firefox (only the plugin part like Flash etc. won´t work anymore and the user gets a message about plugin crashing).