ReadDirectoryChangesW: no immediate FILE_ACTION_MODIFIED on writes, waits for file handle being closed or opened on the file

I am currently trying to use ReadDirectoryChangesW() to listen for filesystem events on Windows in a directory. In general this works, but I've run into an issue: when something just writes to a file (using e.g. WriteFile) no event will be triggered immediately. Instead the event will only be triggered once the file is either closed by the writer, or someone opens a new handle for the same file (assuming that sharing is enabled), but the write itself doesn't cause the event to trigger, even though the file is in fact written. (Also, this doesn't even trigger even after a reasonable wait time - if nobody opens and/or closes a file handle for that file after a write operation, no event will be generated at all.)

The use case is that I want to watch log files that are written and automatically react to new data getting added to the log files. And I'd rather use asynchronous notifications instead of constant polling if that's at all possible. But the current behavior leaves me with no other choice than to poll the log files in that directory regularly.

I've condensed the logic into two example programs: one for processing the notifications (just prints some information on the console), and the other for performing some writes in the file.

The writer program:

#include <iostream>
#include <string>
#include <string_view>
#include <system_error>
#include <stdexcept>
#include <cwchar>
#include <thread>
#include <chrono>
#include <windows.h>

[[noreturn]]
void throwSystemError(DWORD error, std::string message)
{
    std::error_code ec{ static_cast<int>(error), std::system_category() };
    throw std::system_error(ec, std::move(message));
}

int main()
{
    HANDLE fileHandle{ INVALID_HANDLE_VALUE };
    try
    {
        fileHandle = CreateFileW(L"C:\\TestDir\\test.txt", FILE_APPEND_DATA, FILE_SHARE_READ, nullptr, OPEN_ALWAYS,
            FILE_ATTRIBUTE_NORMAL, nullptr);
        if (fileHandle == INVALID_HANDLE_VALUE)
        {
            DWORD error = GetLastError();
            throwSystemError(error, "Could not open \"C:\\TestDir\\test.txt\"");
        }

        for (int i = 0; i < 10; ++i)
        {
            DWORD written = 0;
            BOOL ok = WriteFile(fileHandle, "Entry\n", 6, &written, nullptr);
            if (!ok)
            {
                DWORD error = GetLastError();
                throwSystemError(error, "Could not write to \"C:\\TestDir\\test.txt\"");
            }
            std::cout << "Written new entry to file (bytes written = " << written << ", should be 6)\n" << std::flush;

            /* Change the #if 0 to #if 1 to create a new handle to the file,
             * which in turn will actually cause ReadDirectoryChangesW() to
             * produce an event for the file being modified. But leave as-is,
             * so that only WriteFile() is executed, and the event will only
             * occur once we close the file at the end of the program.
             */
#if 0
            HANDLE temp = CreateFileW(L"C:\\TestDir\\test.txt", FILE_GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
                nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
            if (temp != INVALID_HANDLE_VALUE)
                CloseHandle(temp);
#endif
            
            std::this_thread::sleep_for(std::chrono::seconds{ 2 });
        }

        CloseHandle(fileHandle);
    }
    catch (std::exception& e)
    {
        if (fileHandle != INVALID_HANDLE_VALUE)
            CloseHandle(fileHandle);
        std::cerr << e.what() << std::endl;
        return 1;
    }
    return 0;
}

The watcher program:

#include <iostream>
#include <string>
#include <string_view>
#include <system_error>
#include <stdexcept>
#include <cwchar>
#include <windows.h>

[[noreturn]]
void throwSystemError(DWORD error, std::string message)
{
    std::error_code ec{ static_cast<int>(error), std::system_category() };
    throw std::system_error(ec, std::move(message));
}

int main()
{
    HANDLE dirHandle{ INVALID_HANDLE_VALUE };
    OVERLAPPED ol{};

    try
    {
        dirHandle = CreateFileW(L"C:\\TestDir", GENERIC_READ, FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
            nullptr, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, nullptr);
        if (dirHandle == INVALID_HANDLE_VALUE)
        {
            DWORD error = GetLastError();
            throwSystemError(error, "Could not create directory handle for \"C:\\TestDir\"");
        }

        ol.hEvent = CreateEvent(nullptr, false, false, nullptr);
        if (ol.hEvent == NULL || ol.hEvent == INVALID_HANDLE_VALUE)
        {
            DWORD error = GetLastError();
            throwSystemError(error, "Could not create event object for asynchronous I/O");
        }

        alignas(FILE_NOTIFY_INFORMATION) char buffer[16384];

        while (true)
        {
            DWORD returned{};

            BOOL ok = ReadDirectoryChangesW(dirHandle, buffer, sizeof(buffer), false,
                FILE_NOTIFY_CHANGE_FILE_NAME | FILE_NOTIFY_CHANGE_SIZE | FILE_NOTIFY_CHANGE_LAST_WRITE,
                &returned, &ol, nullptr);
            if (!ok)
            {
                DWORD error = GetLastError();
                throwSystemError(error, "Could not listen for directory changes in \"C:\\TestDir\"");
            }

            DWORD error = ERROR_IO_INCOMPLETE;
            while (error == ERROR_IO_INCOMPLETE)
            {
                DWORD waitResult = WaitForSingleObject(ol.hEvent, INFINITE);
                if (waitResult == WAIT_FAILED)
                {
                    error = GetLastError();
                    throwSystemError(error, "Could not wait for async event for directory changes in \"C:\\TestDir\"");
                }
                if (waitResult == WAIT_OBJECT_0)
                {
                    ok = GetOverlappedResult(dirHandle, &ol, &returned, false);
                    error = GetLastError();
                    if (ok)
                        break;
                    else if (error != ERROR_IO_INCOMPLETE)
                        throwSystemError(error, "Could not wait for async event for directory changes in \"C:\\TestDir\"");
                }
                else
                {
                    error = ERROR_IO_INCOMPLETE;
                }
            }

            auto getPointer = [&](std::size_t offset) {
                if (returned > sizeof(buffer) || offset > returned || (offset + sizeof(FILE_NOTIFY_INFORMATION)) > returned)
                    throw std::runtime_error("Internal error (data exceeds size of the buffer)");
                return reinterpret_cast<FILE_NOTIFY_INFORMATION*>(buffer + offset);
            };

            if (returned == 0)
                continue;

            std::size_t offset = 0;
            DWORD nextOffset = 0;

            do
            {
                auto event = getPointer(offset);

                std::wcout << L"Got change notification for file \"" << std::wstring_view{ event->FileName, event->FileNameLength / sizeof(wchar_t) } << L"\": type " << event->Action << std::endl;

                nextOffset = event->NextEntryOffset;
                offset += nextOffset;
            }
            while (nextOffset > 0);
        }

        if (dirHandle != INVALID_HANDLE_VALUE)
        {
            CancelIo(dirHandle);
            while (!HasOverlappedIoCompleted(&ol))
                (void)SleepEx(1, true);
            if (ol.hEvent != NULL && ol.hEvent != INVALID_HANDLE_VALUE)
                CloseHandle(ol.hEvent);
            CloseHandle(dirHandle);
        }
    }
    catch (std::exception& e)
    {
        if (dirHandle != INVALID_HANDLE_VALUE)
        {
            CancelIo(dirHandle);
            while (!HasOverlappedIoCompleted(&ol))
                (void)SleepEx(1, true);
            if (ol.hEvent != NULL && ol.hEvent != INVALID_HANDLE_VALUE)
                CloseHandle(ol.hEvent);
            CloseHandle(dirHandle);
        }

        std::cerr << e.what() << std::endl;
        return 1;
    }
    return 0;
}

My questions would be:

Am I doing something wrong here so that this doesn't work as expected?
If however this is the behavior of Windows, is there any way to achieve what I am trying to do? Such as calling some other WinApi function that forces some state to by synchronized or something? I do also control the writer process, so I could add some code there, but the workaround I've found of opening an additional handle to the file and then immediately closing that again doesn't sit right with me. (Also, ideally, I'd like to avoid changing the writer if at all possible, because from an abstraction point of view it doesn't make much sense to me that I should have to change that part of the software.)

Solution

As @RaymondChen said in comment,

ReadDirectoryChangesW reads changes to the directory. But writes do not update the directory until the handle is closed. (If you type "dir", you can see that the file size is unchanged, so the directory has not been updated.)

WriteFile also states,

When writing to a file, the last write time is not fully updated until all handles used for writing have been closed. Therefore, to ensure an accurate last write time, close the file handle immediately after writing to the file.