I have one C code app. which i was building using MS-VS2005. I had one output data buffer which was being allocated dynamically using malloc.
For some test cases, the memory size which was being malloc'd was falling short than the the actual output size in bytes which was generated. That larger sized output was written into the smaller sized buffer causing buffer overflow. As a result of which the test-run was crashing with MSVS-2005 showing up a window "Heap corruption ...."
I knew it had to do with some dynamic memory allocation, but i took long time to actually find the root cause, as i did not doubt the memory allocation because i was allocating large enough size necessary for the output. But one particular test case was generating more output than what i had calculated, hence the resulting crash.
My question is:
1.) What tools i can use to detect such dynamic memory buffer over-flow conditions. Can they also help detect any buffer overflow conditions(irrespective of whether the buffer/array is on heap, stack, global memory area)?
2.) Will memory leak tools(like say Purify) or code analysis tools like lint, klocworks would have helped in particular case? I believe they have to be run time analysis tools.
Thank you.
-AD.
One solution, which I first encountered in the book Writing Solid Code, is to "wrap" the malloc() API with diagnostic code.
First, the diagnostic malloc() arranges to allocate additional bytes for a trailing sentinel. For example, an additional four bytes following the allocated memory are reserved and contain the characters 'FINE'.
Later, when the pointer from malloc() is passed to free(), a corresponding diagnostic version of free() is called. Before calling the standard implementation of free() and relinquishing the memory, the trailing sentinel is verified; it should be unmodified. If the sentinel is modified, then the block pointer has been misused at some point subsequent to being returned from the diagnostic malloc().
There are advantages of using a memory protection guard page rather than a sentinel pattern for detecting buffer overflows. In particular, with a pattern-based method, the illegal memory access is detected only after the fact. Only illegal writes are detected by the sentinel pattern method. The memory protection method catches both illegal reads and writes, and they are detected immediately as they occur.
Diagnostic wrapper functions for malloc() can also address other misuses of malloc(), such as multiple calls to free() for same memory block. Also, realloc() can be modified to always move blocks when executed in a debugging environment, to test the callers of realloc().
In particular, the diagnostic wrappers may record all of the blocks allocated and freed, and report on memory leaks when the program exits. Memory leaks are blocks which are allocated by not freed during the program execution.
When wrapping the malloc() API, one must wrap all of the related functions, including calloc(), realloc(), and strdup().
The typical way of wrapping these functions is via preprocessor macros:
#define malloc(s) diagnostic_malloc(s, __FILE__, __LINE__) /* etc.... */
If the need arises to code a call to the standard implementation (for example, the allocated block will be passed to a third-party, binary-only library which expects to free the block using the standard free() implementation, the original function names can be accessed rather than the preprocessor macro by using (malloc)(s) -- that is, place parentheses around the function name.