How to verify out-of-memory behaviour of a program

Question

The software I'm working on has quite a lot of lines handling out-of-memory situations, like this

/* Leave object->data.cue_sheet.tracks untouched if realloc fails */
FLAC__StreamMetadata_CueSheet_Track *tmpptr;
if ((tmpptr = realloc(object->data.cue_sheet.tracks, new_size)) == NULL)
    return false;
object->data.cue_sheet.tracks = tmpptr;

In the past, bugs (mostly memory leaks) in code like this were found, but debugging and testing this code is quite hard, because I cannot seem to find a way to simulate out-of-memory behaviour on my PC. Valgrind does not seem to have functionality for this, and ulimit does not work with tools like address sanitizer. However, this software (libFLAC) is used in embedded environment, so I'd like to verify this behaviour to a reasonable extent.

Is there a way to trigger out-of-memory handling?

Answer 1

I've implemented this with fuzzing. This makes it nicely reproduceable as well. The library fuzzer sets a number at the start of each fuzz attempt and all malloc/realloc/calloc calls are intercepted. In case the intercepted malloc/realloc/calloc is the nth one in that attempt, it is simulated to fail.

In that way, the fuzzer logic can be used to find interesting combinations of settings and a certain allocation failing.

Here's the bit of code I implemented and a small additional bit here

An excerpt from the code in the library that is fuzzed:

#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
extern int alloc_check_threshold, alloc_check_counter;

static inline int alloc_check() {
    if(alloc_check_threshold == INT32_MAX)
        return 0;
    else if(alloc_check_counter++ == alloc_check_threshold)
        return 1;
    else
        return 0;
}

static inline void *realloc_(void *ptr, size_t size)
{
    /* Fail if requested */
    if(alloc_check())
        return NULL;
    return realloc(ptr, size);
}
#else
#define realloc_ realloc
#endif

and an excerpt in the fuzzer

extern int alloc_check_threshold, alloc_check_counter;
int alloc_check_threshold = INT32_MAX, alloc_check_counter = 0;

[...]

alloc_check_threshold = data[2];
alloc_check_counter = 0;