Search code examples
calloca

C alloca function - what happens when too much memory is tried to be allocated

In C the alloca() function allocates memory on the stackframe of the caller of alloca().

What happens when you try to allocate a huge number of bytes that it cannot possibly allocate?

  • Does it allocate as many bytes as it can until the stack meets the heap segment?

  • Or does it allocate nothing at all?

    alloca( 100000000000000000000 );

The manual mentions:

The alloca() function returns a pointer to the beginning of the allocated space. If the allocation causes stack overflow, program behavior is undefined.

I understand the behaviour is undefined. But there must be more to say than that:

  • What does it return, a pointer to the first byte after the top of the stack before main was called?
  • After alloca() returned is the stack pointer different then what it was before alloca() was called?

Does anyone have some more info about this?

Solution

  • The GNU libc implementation of alloca() reads:

    # define alloca(size)   __builtin_alloca (size)

    It uses a compiler built-in, thus it depends entirely on how it's been implemented by the compiler. More specifically, it depends on how the stack is handled, which happens to be a a machine and ABI-dependent data structure.

    Let's examine a concrete case. On my machine, this is the assembly for alloca(100000000000L):

    0e9b:  movabsq $-100000000016, %rax ; * Loads (size + 16) into rax.
0ea5:  addq   %rax, %rsp            ; * Adds it to the top of the stack.
0ea8:  movq   %rsp, -48(%rbp)       ; * Temporarily stores it.
0eac:  movq   -48(%rbp), %rax       ; * These five instructions round the
0eb0:  addq   $15, %rax             ;   value stored to the next multiple
0eb4:  shrq   $4, %rax              ;   of 0x10 by doing:
0eb8:  shlq   $4, %rax              ;   rax = ((rax+15) >> 4) << 4
0ebc:  movq   %rax, -48(%rbp)       ;   and storing it again in the stack.
0ec0:  movq   -48(%rbp), %rax       ; * Reads the rounded value and copies
0ec4:  movq   %rax, -24(%rbp)       ;   it on the previous stack position.

    Compiled using gcc-4.2 -g test.c -o test from the following program:

    With something to refer to, your questions can now be answered:

    Does it allocate as many bytes as it can until the stack meets the heap segment?

    It just blindly grows the stack by the requested number of bytes. No bounds checking is performed at all, so both the stack pointer and the returned value will probably now be in a illegal location. Attempting to read/write from the returned value (or push onto the stack) will result on a SIGSEGV.

    What does it return, a pointer to the first byte after the top of the stack before main was called?

    It returns a pointer to the first byte of allocated memory.

    After alloca() returned is the stack pointer different then what it was before alloca() was called?

    Yes, see explanation above. Also, when the function that called alloca returns, the stack will be restored to previous frame and it will be usable again.