How can we know the minimum stack size needed by a program launched with exec()?

In an attempt to avoid stack clash attacks against a program, we tried to set a limit on the stack size with setrlimit(RLIMIT_STACK) to about 2 MB.

This limit is fine for our program's own internal needs, but we then noticed that attempts to exec() external programs began to fail on some systems with this new limit. One system we investigated using the test program below seems to have a minimum stack size for exec()'d programs of a bit over 4 MiB.

My question is, how can we know the safe minimum value for the stack size on a given system, so that exec() will not fail?

We don't want to just raise this until things stop failing on all the systems we currently test against, since that is likely to cause failures in the future as the program is ported to newer system types with higher minimum requirements.

The C test program below is written in terms of system(), but the lower-level symptom is a failure in the execl() syscall. Depending on the host OS you test on, you either get errno == E2BIG or a segfault in the called program when you give the called program too little stack space to start up.

Build with:

$ CFLAGS="-std=c99 -D_POSIX_C_SOURCE=200809" make stacklim

This question is tangentially-related to "To check the E2BIG error condition in exec", but our actual question is different: we're interested in the potential portability problem setting this limit causes.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <unistd.h>

static enum try_code {
    raise_minimum,
    lower_maximum,
    failed_to_launch
}
try_stack_limit(rlim_t try)
{
    // Update the stack limit to the given value
    struct rlimit x;
    getrlimit(RLIMIT_STACK, &x);
    static int first_time = 1;
    if (first_time) {
        first_time = 0;
        printf("The default stack limit here is %llu bytes.\n", x.rlim_cur);
    }
    x.rlim_cur = try;
    setrlimit(RLIMIT_STACK, &x);

    // Test the limit with a do-nothing shell launch
    int status = system("exit");
    switch (status) {
        case 0: return lower_maximum;

        case -1:
            perror("Failed to start shell");
            return failed_to_launch;

        default:
            if (WIFEXITED(status) && WEXITSTATUS(status) == 127) {
                // system() couldn't run /bin/sh, so assume it was
                // because we set the stack limit too low.
                return raise_minimum;
            }
            else if (WIFSIGNALED(status)) {
                fprintf(stderr, "system() failed with signal %s.\n",
                        strsignal(WTERMSIG(status)));
                return failed_to_launch;
            }
            else {
                fprintf(stderr, "system() failed: %d.\n", status);
                return failed_to_launch;
            }
    }
}

int main(void)
{
    extern char **environ;
    size_t etot = 0;
    for (size_t i = 0; environ[i]; ++i) {
        etot += strlen(environ[i]) + 1;
    }
    printf("Environment size = %lu\n", etot + sizeof(char*));

    size_t tries = 0;
    rlim_t try = 1 * 1000 * 1000, min = 0, max = 0;
    while (1) {
        enum try_code code = try_stack_limit(try);
        switch (code) {
            case lower_maximum:
                // Call succeded, so lower max and try a lower limit.
                ++tries;
                max = try;
                printf("Lowered max to %llu bytes.\n", max);
                try = min + ((max - min) / 2);
                break;

            case failed_to_launch:
                if (tries == 0) {
                    // Our first try failed, so there may be a bug in
                    // the system() call.  Stop immediately.
                    return 2;
                }
                // Else, consider it a failure of the new limit, and
                // assume we need to limit it.

            case raise_minimum:
                // Call failed, so raise minimum and try a higher limit.
                ++tries;
                min = try > min ? try : min;
                rlim_t next = max ?
                        min + ((max - min) / 2) :
                        try * 2;
                if (next == try) {
                    printf("Min stack size here for exec is %llu.\n", max);
                    return 0;
                }
                else {
                    printf("Raising limit from %llu to %llu.\n", try, next);
                    try = next;
                }
                break;

            default:
                return 1;
        }
    }
}

Solution

Your program was launched successfully, therefore your program was implicitly given the correct stack size for launching other programs in turn: during your program's startup, get the current limit before you set the new lower limit:

struct rlimit g_default_stack_limit;  /* put in global scope */
getrlimit(RLIMIT_STACK, &g_default_stack_limit);

struct rlimit our_stack_limit;
memcpy(&our_stack_limit, &g_default_stack_limit, sizeof(our_stack_limit));
our_stack_limit.rlim_cur = 2000000;   /* some lower value */
setrlimit(RLIMIT_STACK, &our_stack_limit);

Then restore that initial value before launching an external program, and reapply the new limit after either the child fork() is created or a synchronous call of the program (e.g. via system()) exits:

struct rlimit our_stack_limit;
getrlimit(RLIMIT_STACK, &our_stack_limit);
setrlimit(RLIMIT_STACK, &g_default_stack_limit);

if (system(...) == 0) {
    ....
}

setrlimit(RLIMIT_STACK, &our_stack_limit);

This initial value may be the operating system's default or it may be a limit set by the program that called your program. Either way, it is almost certainly the correct starting value to pass on to programs your program calls in turn.