I am making a library which gets injected into processes and redefines some of functions, such as open(2) to do some tasks before calling real open(2). My library would call mmap(2). As open(2) is async safe, is it possible that someone using the library who called open(2) in signal handler and where my library also added a call to mmap(2) could make his call to open(2) go wrong?
UPDATED QUESTION:
void handle_sigint(int sig)
{
int fd = open(“file”, O_RDWR, 0666);
void *base = mmap(NULL, 20, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}
In the above function, will the call to open(2) be effected because I called mmap(2)?
Yes.
If your substitute open() calls a function that is not async-signal-safe, then it is not safe for a signal handler to call your function. That it has the same name and signature as a standard function that is async-signal-safe is irrelevant. That it calls the replaced function or other async-signal-safe functions is irrelevant. That the prospective signal handler's call to a function that is not async-signal-safe would not be a direct one is irrelevant.
In response to the question update: in the event that the function presented in the question is called as a signal handler, it has undefined behavior on account of its call to mmap(). The details of that UB cannot be predicted, at least not from the relevant standards. That's what "undefined" means. There is no reason whatever to suppose that the actual and apparent effects of the open() call would be somehow protected from interference. Nor the general signal-handling mechanism. Nor anything else in the program.
The further you get from the locus of the UB, the less likely is any noticeable effect, and the more likely the OS is to contain it, but UB is not something to mess with. In principle, it may manifest as any behavior or behaviors within the power of your computer to produce, such as wiping your disk, turning off your CPU fan, or mailing your password to hackers.