Despite a careful read of the related standard documentation, I can't understand what's the expected behavior in POSIX compliant systems when a open system call is invoked with flags including O_CREAT|O_DIRECTORY.
The standard specifies that
If O_CREAT and O_DIRECTORY are set and the requested access mode is neither O_WRONLY nor O_RDWR, the result is unspecified.
However it does not specify the behavior of the system with neither (O_CREAT|O_DIRECTORY|O_WRONLY) nor (O_CREAT|O_DIRECTORY|O_RDWR). Indeed (as far as I can understand) the behavior on EISDIR only apply to existing directories.
In the section related to O_CREATE, the standard specifies that, when the named file does not exist,
if O_DIRECTORY is not set the file shall be created as a regular file; [...]
but again it does not specify what will happen if O_DIRECTORY is set too.
I've looked the manual pages of both NetBSD (which notoriously cares a lot about POSIX compliance) and Linux (which is a widely used system, despite not actually a POSIX one) but I can't find any clarification.
Is it correct to say that the use of both flags is unspecified? And if so, what's the most common behavior?
Is open(name, O_CREAT|O_DIRECTORY, mode) equivalent to mkdir on any POSIX compliant OS?
NetBSD itself contains the following in vn_open:
if ((fmode & (O_CREAT | O_DIRECTORY)) == (O_CREAT | O_DIRECTORY))
return EINVAL;
so any combination with these 2 is straight up rejected.
In Linux, it's a little bit more hairy, but any trivial test will show you that the directory is not created either, but you can end up with a file.
For kicks, I also checked FreeBSD, which never ends up creating anything with O_DIRECTORY in the first place.
If what you are looking for is a mkdir which gives you back the fd, I'm afraid there is nothing of the sort. On the other hand, you should be able to safely open with O_DIRECTORY anything you mkdir'ed.