I'm trying to understand what is happening behind the scenes with this code. This was asked at a final exam of an Intro to OS course I'm taking. As I understand it, when returning from kernel mode to user mode, the system checks if there are any pending signals (by examining the signal vector) and tends to them. So as the program returns from the kill syscall the OS sees that SIGUSR1 is pending and invokes the handler. By that logic, the handler should print "stack" in an infinite loop, but when running this code it actually prints "stackoverflow" in an infinite loop. Why is this happening? Thanks in advance.
void handler(int signo) {
printf("stack");
kill(getpid(), SIGUSR1);
printf("overflow\n");
}
int main() {
struct sigaction act;
act.sa_handler = &handler;
sigaction(SIGUSR1, &act, NULL);
kill(getpid(), SIGUSR1);
return 0;
}
You actually have undefined behavior here, as you're calling sigaction with an incompletely initialized struct sigaction object. So depending on what values happen to be in the sa_flags and sa_mask fields, a variety of different things might happen.
Some of these would not block SIGUSR1 while the signal handler is running, which would mean that a new signal handler would run immediately when the first calls kill (so before the first handler returns and pops its stack frame). So you end up with many recursive handler stack frames on the stack (and outputs of 'stack') until it overflows.
Other combos would block the signal so it would not immediately trigger a second signal handler. Instead the signal would be "pending" until the first signal handler returns.