I am implementing a cross platform scripting language for our product. There is a requirement to detect and properly handle stack overflow condition in language VM. Before you jump in and say make sure there is no stack overflow in the first place, re-read my first sentence - this is a scripting language and end users may write incorrect programs in this language, which may overflow the stack via for example endless recursion.
Now I know how to detect and recover from stack overflow in Windows (see http://support.microsoft.com/kb/315937). However I am unable to find any solution for Mac OS X.
The VM is implemented in C++: MSVC++ on Windows, GCC on Mac OS X.
Ideally the mechanism must be based on UNIX capabilities since we also plan to port to Linux.
Thanks.
OCaml has the same constraints as you ("scripting" language where the programmer may cause a stack overflow). Its native compiler uses the system stack for function calls -- as you do -- and it handles stack overflows (materializing them as exceptions).
If you do not receive a more explicit answer, I suggest you look at how it's done in the OCaml sources.
~/ppc $ cat >> t.ml
let rec f x = (f x) + (f x) ;;
f 0 ;;
~/ppc $ ocamlopt t.ml
~/ppc $ ./a.out
Fatal error: exception Stack_overflow
The above is on Mac OS X Leopard. Search for #ifdef HAS_STACK_OVERFLOW_DETECTION in the source files.