I'm using the pthread.h library in glibc-2.27 and when my process calls pthread_create() eighteen times or more (it's supposed to be a heavy multi-threaded application) the process is aborted with the error message:
*** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)
I did some strace as part of my debugging ritual and I found the reason. Apparently all implicit calls for mmap() as part of the pthread_create() looks like this:
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f6de43fa000
One can notice the MAP_STACK flag which indicates:
Allocate the mapping at an address suitable for a process or thread stack. This flag is currently a no-op, but is used in the glibc threading implementation so that if some architectures require special treatment for stack allocations, support can later be transparently implemented for glibc.
(man mmap on my system - Ubuntu 18.04 LTS)
It is possible to configure the pthread_create call not to do this? or maybe use brk or something else to increase the data segment automatically?
Thanks for any help!
It is possible to configure the pthread_create call not to do this?
pthread_create() needs to allocate space for the thread's stack, otherwise the thread cannot run -- not even with an empty thread function. That's what the mmap you're seeing is for. It is not possible to do without.
or maybe use brk or something else to increase the data segment automatically?
If you have the time and skill to write your own thread library, then do have a go and let us know what happens. Otherwise, no, the details of how pthread_create() reserves space for the new thread's stack is not configurable in any implementation I know.
And that does not matter anyway, because the mmap() call is not the problem. If a syscall has an unrecoverable failure then that's a failure of the kernel, and you get a kernel panic, not an application crash. GNU C's stack-smashing detection happens in userspace. The functions to which it applies therefore do not appear in your strace output, which traces only system calls.
It might be useful to you have a better understanding of stack smashing and GNU's defense against it. Dr. Dobb's ran a nice article on just that several years ago, and it is still well worth a read. The bottom line, though, is that stack smashing happens when a function implementation misbehaves by overwriting the part of its stack frame that contains its return address. Unless you've got some inline assembly going on, the smashing almost surely arises from one of your own functions overrunning the bounds of one of its local variables. It is detected when that function tries to return, by tooling in the function epilogue that serves that purpose.