Automatic file buffer flushing when file opened by another program only if other program not already running (macOS)

I've run into a weird behavior on macOS that I started investigating due to an accidental race condition that would occur when telling the OS to open an RTF file I'd just processed, but hadn't closed or explicitly flushed.

If the file handler (Word, TextEdit, whatever) was not already open, then calling system("open test.rtf") would open the file just fine and the file would be complete.

However, if the file handler was already open, then calling system("open test.rtf") would result in an error message that the file was corrupted or truncated (because the buffers weren't completely flushed).

The obvious fix is to fflush() and/or fclose() my file before opening it. However, I am more interested in the underlying interaction between my program's runtime and macOS. My question is this: how and why does the file handler's running/not-running state affect whether my buffer is flushed?

(It isn't simply a matter of the time it takes to open the program -- I added a sleep delay in the version that doesn't explicitly flush the buffers and it makes no difference.)

Unflushed version (works only if the file handler is not already running):

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <unistd.h>

int main(void) {
    FILE *fin  = fopen("src.rtf", "rb");
    FILE *fout = fopen("test.rtf", "wb");
    int c;

    assert(fin && fout);
    while ((c=fgetc(fin)) != EOF) fputc(c, fout);

    sleep(3);
    system("open test.rtf");

    fclose(fin);
    fclose(fout);
    
    return 0;
}

Explicitly flushed version (works all the time):

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

int main(void) {
    FILE *fin  = fopen("src.rtf", "rb");
    FILE *fout = fopen("test.rtf", "wb");
    int c;

    assert(fin && fout);
    while ((c=fgetc(fin)) != EOF) fputc(c, fout);

    fflush(fout);
    
    system("open test.rtf");

    fclose(fin);
    fclose(fout);
    
    return 0;
}

The sample RTF file I'm using is here: https://pastebin.com/mXLk85G1

Solution

Okay, so, my best guess is this:

system()→fork()→exec()→/bin/sh

/bin/sh processes its arguments and dispatches the command:

fork()→exec()→open

/usr/bin/open processes its arguments, looks up the file handler via LaunchServices, and tries to open the file with the appurtenant program.

Here's the pure guessing:

Already Running: /usr/bin/open uses IPC to tell the already-running application to try to open the file. The application opens a new file descriptor in its existing file descriptor table and reads it in, getting the truncated version because the original stream hasn't been fflush()'d or fclose()'d yet.
Not Yet Running: /usr/bin/open sees that the application is not running yet, and launches it via fork()→exec(). This means that the application still has the original fd table from the original program. The Cocoa runtime checks the fd table, sees its already open for writing, and closes it before re-opening for reading, causing the output buffer to be flushed.

I have verified that closing the file in a child of fork() will result in the output buffer being flushed. The following will work consistently:

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <unistd.h>

int main(void) {
    FILE *fin  = fopen("src.rtf", "rb");
    FILE *fout = fopen("test.rtf", "wb");
    int c;
    int pid, wpid, wstat; 

    assert(fin && fout);
    while ((c=fgetc(fin)) != EOF) fputc(c, fout);

    pid = fork();
    if (!pid) {
        fclose(fout);
        fout = NULL;
    } else {
        wpid = waitpid(pid, &wstat, WUNTRACED);
        execl("/bin/sh", "sh", "-c", "open test.rtf", (char *)0);
    }

    if (fin)  fclose(fin); 
    if (fout) fclose(fout); 
    
    return 0;
}

However, there is one major problem with my theory: While the fd table could potentially carry across fork() and exec() all day long, process memory, including the FILE*, will be destroyed when exec() overwrites the image.

After pursuing this further, I found that opening a Cocoa file handler causes launchd to launch /sbin/filecoordinationd, a daemon to "coordinate access to files." https://www.unix.com/man-page/osx/8/filecoordinationd/. And, indeed, TextEdit registers as an NSFilePresenterProxy. macOS has an entire underlying file access regime to watch for file changes and make sure files accessed by different processes are kept in a good state. It would make sense that, once TextEdit registers as an NSFilePresenter, invoking /sbin/filecoordinationd, the daemon would ensure any buffers it knew were already open would get to a good state.

But how would it do that with my program, which doesn't use Cocoa and isn't registered as an NSFile-anything? The most likely answer would be that the mechanics of file coordination for NS-classes is implemented in libSystem.dylib, which also serves as the system C library. The macOS system C library likely comes with the ability baked-in to have the operating system make process runtime buffers flush.

So why doesn't it do this when the Cocoa application is already running? It probably doesn't know that it should. If TextEdit didn't open the file, and the process with the file open doesn't register with NSFile..., and TextEdit doesn't inherit a file descriptor table, nothing in the Cocoa ecosystem would have any idea that it should tell /sbin/filecoordinationd to make sure output buffers to the file are flushed.

This seems like the best working theory and as close as I'm going to get to an answer without input from a macOS engineer or access to the /usr/bin/open and /sbin/filecoordinationd source code.

Automatic file buffer flushing when file opened by another program *only if* other program not already running (macOS)

Automatic file buffer flushing when file opened by another program only if other program not already running (macOS)