In our product we use a malloc implementation that relies exclusively on mmap for memory allocation. We also do a fair use of allocaing. We've just encountered a problem where mmap will allocate regions that should be reserved to stack space (thus causing very bad things to happen when one of our larger alloca's spills into the malloc'd region).
The limitation of our process' allocation is our VM address space, not physical memory. We've watched the /proc/*/maps file as the process runs and watched malloc eat up any available address space. It eventually resorts to allocating addresses within the stacks rlimit-set range, and eventually a large alloca stretches into it.
We've tried to work around it by allocaing our entire stack limit at startup, but that hasn't proved stable across platforms (it segfaults trying to access the allocad memory on my 2.4 dev box, while it works on the 2.6 production machine).
Is there any way to actually reserve the address space? What else can be done?
This is apparently an recently documented security vulnerability used in a privilege escalation exploit. Allegedly newer kernel versions will be patched.