The standard way would be the following:
if (ptrace(PTRACE_TRACEME, 0, NULL, 0) == -1)
printf("traced!\n");
In this case, ptrace returns an error if the current process is traced (e.g., running it with GDB or attaching to it).
But there is a serious problem with this: if the call returns successfully, GDB may not attach to it later. Which is a problem since I'm not trying to implement anti-debug stuff. My purpose is to emit an 'int 3' when a condition is met (e.g., an assert fails) and GDB is running (otherwise I get a SIGTRAP which stops the application).
Disabling SIGTRAP and emitting an 'int 3' every time is not a good solution because the application I'm testing might be using SIGTRAP for some other purpose (in which case I'm still screwed, so it wouldn't matter, but it's the principle of the thing :))
You could fork a child which would try to PTRACE_ATTACH its parent (and then detach if necessary) and communicates the result back. It does seem a bit inelegant though.
As you mention, this is quite costly. I guess it's not too bad if assertions fail irregularly. Perhaps it'd be worthwhile keeping a single long-running child around to do this - share two pipes between the parent and the child, child does its check when it reads a byte and then sends a byte back with the status.