Consistency of two C FILE* streams on a single file

Question

I need to implement a simple "spill to disk" layer for large volume of data coming off a network socket. I was hoping to have two C FILE* streams, one used by a background thread writing to the file, one used by a front end thread reading it.

The two streams are so one thread can be writing at one offset, while the other is reading elsewhere - without taking a lock and blocking the other thread.

There will be a paging mechanism so the reads/writes are at random access locations - not necessarily sequential.

One more caveat, this needs to work on Windows and Linux.

The question: after the fwrite to the first stream has returned, is that written data guaranteed to be immediately visible to an fread on the second stream?

If not, what other options might I consider?

---

So Posix pread/pwrite functions turned out to be what I needed. Here's a version for Win32:

size_t pread64(int fd, void* buf, size_t nbytes, __int64 offset)
{
    OVERLAPPED ovl;
    memset(&ovl, 0, sizeof(ovl));
    *((__int64*)&ovl.Offset)=offset;

    DWORD nBytesRead;
    if (!ReadFile((HANDLE)_get_osfhandle(fd), buf, nbytes, &nBytesRead, &ovl))
        return -1;

    return nBytesRead;
}

size_t pwrite64(int fd, void* buf, size_t nbytes, __int64 offset)
{
    OVERLAPPED ovl;
    memset(&ovl, 0, sizeof(ovl));
    *((__int64*)&ovl.Offset)=offset;

    DWORD nBytesWritten;
    if (!WriteFile((HANDLE)_get_osfhandle(fd), buf, nbytes, &nBytesWritten, &ovl))
        return -1;

    return nBytesWritten;
}

(And thank you everyone for input on this - much appreciated).

Answer 1

This definitely will not give you the semantics you want. If you disabled buffering, it might be reasonable to expect it to work, but I still don't think there are any guarantees. Stdio/FILE is really not the right tool for specialized IO needs like this.

The POSIX way to do what you want is with file descriptors and the pread/pwrite functions. I suspect there's a Windows way (or you could emulate them based on some other underlying Windows primitive) but I don't know it.

Also Ben's suggestion of using memory-mapped IO is a very good one, assuming the file fits in your address space.