I'm using Google Test framework.
Now I need to test my program, to see whether or not it could fork() and exec() another child process with expected cli arguments when some conditions.
But fork() and exec() are sys calls, not common function calls.
I can NOT simply capture the calls with mock objects.
How to do it?
Use dependency injection and inject the syscall function in your production code through an abstraction layer (refer to https://stackoverflow.com/a/54148046/7717227 )
I recommend this approach because it is more in the spirit of "unit" testing, since you wouldn't spawn a new process.
You could also test the result that you expect to see. When fork fails, it returns -1, which you can propagate upwards and check in your unit test. Alternatively, you could use another syscall, waitpid, to check if a child process is running. See: How to check if all child processes ended?
Keep in mind that the application you exec should have its own unit tests, and you should not be testing that here. There should also be minimal logic between the fork and exec - any error paths would ideally be validated in the parent process before the fork. Potential system-level issues (such as changing users or reaching a process limit) are beyond the scope of unit testing and belong in an integration or system-level test.