The way I understand it, segmented stacks are built with compiler support so that whenever a function running on the segmented stack calls another function, if first checks whether the stack has enough space for the stack frame for that new function. And if it doesn't another segmented stack is attached and code branches to that function.
But does this work if say for example I have a fiber running and I call another function from another shared (or compiled into non shared object file) library that was not compiled with the -fsplit-stack option? How do the functions in that library know that they would have to check to see if the segmented stack has enough space in the segmented stack to continue?
Only interested in clang and gcc implementations (and in particular with boost context), thanks!
I'm going to grab back at a piece of documentation that I remember seeing at an earlier question about this subject:
Backward compatibility
We want to be able to use split stack programs on systems with prebuilt libraries compiled without split stacks. This means that we need to ensure that there is sufficient stack space before calling any such function.
Each object file compiled in split stack mode will be annotated to indicate that the functions use split stacks. This should probably be annotated with a note but there is no general support for creating arbitrary notes in GNU
as. Therefore, each object file compiled in split stack mode will have an empty section with a special name: .note.GNU-split-stack. If an object file compiled in split stack mode includes some functions with theno_split_stackattribute, then the object file will also have a.note.GNU-no-split-stack section. This will tell the linker that some functions may not have the expected split stack prologue.
[...]
For calls from split-stack code to non-split-stack code, the linker will change the initial instructions in the split-stack (caller) function. This means that the linker will have to have special knowledge of the instructions that the compiler emits. The effect of the changes will be to increase the required framesize by a number large enough to reasonably work for a non-split-stack. This will be a target dependent number; the default will be something like 64K. Note that this large stack will be released when the split-stack function returns. Note that I'm disregarding the case of split-stack code in a shared library calling non-split-stack code in the main executable; that seems like an unlikely problem.
I specifically remember the list (italicized) caveat - I don't remember it was me who highlighted it or someone else. The keyword in that discussion was about "callbacks".