Suppose you're compiling a functional language to portable C, and suppose also that for various reasons you want precise rather than conservative garbage collection. There is no portable way (perhaps no way at all in the general case) for the garbage collector to figure out what is and isn't a pointer on the C stack. It seems to me there are two solutions to this problem:
Shadow stack. Make each C function maintain bookkeeping information about what is and isn't a pointer. This is the approach recommended by e.g. LLVM.
Take advantage of the fact that you are compiling a functional language, which means mainline code has no side effects. When the allocator detects out of memory, instead of calling the garbage collector itself, it aborts the current operation with a longjmp back to the main loop, which calls the garbage collector (in a context where the set of variables that may contain pointers is known in advance) then restarts the operation.
It seems to me that, if you are dealing with a pure functional language where the second approach is applicable, it must be more efficient than the first approach, as well as easier to mix with handwritten C.
Are there any problems I'm overlooking? Any references to existing discussion or implementations of this technique?
I think the most obvious thing you haven't described is how to handle persistent out-of-memory. As you've described it, suppose I have an operation that uses more memory than is available. Eventually I reach:
That is, an infinite loop. So at the least you want some sort of generational garbage collection that will allow you to detect the loop and exit.