I would like to mock system calls such as malloc/file-open for unit testing my code without modifying the production code. Also, creating wrappers for these system calls in source code is not feasible and hence that option is ruled out.
Any inputs/solution for the will be helpful.
Mocking system calls is problematic, since the same functions might also be used by gtest and gmock themselves. And, since wrapping them in the source code is also not an option, there are probably not too many possibilities left.
One thing you could try is to use the preprocessor to replace these calls when compiling your source code file. It is not standard compliant, but will typically work. Let's assume the code you need to test is in file foo.cpp. This file foo.cpp includes a.h and b.h and, for malloc, <cstdlib>.
What you want to do is to #define malloc malloc_mock, but to make it work (at least likely - as I said it is a non-compliant hack), you have to do it in the following way, in your file foo_test.cpp:
#include <a.h> // before malloc is re-defined
#include <b.h> // also before re-defining malloc
#include <cstdlib> // and again
// Now all files are included that foo.cpp also includes.
// Let's hope every file has an include-guard in place ...
void* malloc_mock (size_t size);
#define malloc malloc_mock
// now, in the code of foo.cpp, all calls to malloc are substituted:
#include "foo.cpp" // include guards protect a.h etc. from being affected
#undef malloc
... your tests come here
Ugly? Sure. But, the restrictions came from you, so don't ask for something beautiful.
Just a side note: In the special case of mocking malloc I would assume you are trying to implement some memory leak checking - if so, I recommend not to mock it, but instead run your unit-tests under valgrind...