If I want to rename A to B, but only if B doesn't exist, the naive thing would be checking if B exists (with access("B", F_OK) or something like that), and if it doesn't proceeding with rename. Unfortunately this opens a window during which some other process might decide to create B, and then it gets overwritten - and even worse there's no indication that something like that ever happened.
Other file system access functions don't suffer from this - open has O_EXCL (so copying files is safe), and recently Linux got an entire family of *at syscalls that protect against most other race conditions - but not this particular one (renameat exists, but protects against an entirely different problem).
So does it have a solution?
As of Linux kernel 3.15 (released in June 2014), this can be done with syscall(__NR_renameat2, AT_FDCWD, "source-file", AT_FDCWD, "dest-file", RENAME_NOREPLACE) (include <syscall.h>, <fcntl.h> and <linux/fs.h>).
This is better than link(), because there is never a point where both filenames exist simultaneously (in particular, with link(), a precisely-timed power outage could cause both names to remain forever).
glibc 2.28 (released in August 2018) adds a renameat2() wrapper, so you can use that instead of syscall.h and linux/fs.h (though you'll most likely need <stdio.h> and #define __GNU_SOURCE instead).
For more details, see http://man7.org/linux/man-pages/man2/rename.2.html (though it does, as of writing, not know that glibc now has a renameat2 wrapper).