Search code examples
c++dllcomipcmemory-mapped-files

OpenFileMapping() can't find MMF created in COM DLL


Context

I'm trying to send image data at runtime from a C++ .exe application to a DirectShow source filter implemented as a COM DLL. To do this, I'm trying to use memory-mapped files to perform IPC using a slightly modified version of the code taken from here. All clients and servers use the same ipc.c and ipc.h.

hFile = INVALID_HANDLE_VALUE, and I allow an external string to be used to set the lpName.

Server

I set the DLL debugger to Command -> lync.exe and Debugger Type -> Mixed, allowing me to debug the actual use of the filter. Server creation (static instance with function scope) succeeds and returns a valid handle.

HRESULT CVSourceStream::FillBuffer(IMediaSample *pms) {
    static osIPC::Server server(IPC_SERVER_ADDR); // filename shared by client/server   
    ... // set timestamp on sample
    BYTE *pData;
    pms->GetPointer(&pData);
    server.read(&pData, IPC_BLOCK_SIZE);
    return S_OK;
}

Client

However, when I attempt to open the file mapping when creating the osIPC::Client instance in my application, the OpenFileMapping() call fails by returning NULL, with the last error code being ERROR_FILE_NOT_FOUND.

osIPC::Client client(IPC_SERVER_ADDR); // class instance member

Testing

To test, I wrote a lightweight client/server pair of Visual Studio solutions that each have the corresponding types of initialization (non-static/static). Each solution is basically just a main() that creates the osIPC::Client or osIPC::Server then passes byte buffers across. Here, the OpenFileMapping() call succeeded.

Running my application in conjunction with the lightweight server succeeded, but combining the DLL with the lightweight client failed, leading me to suspect the filter was causing issues.

I tried appending Global\\ to the file name passed to the CreateFileMapping() call to use the global namespace with no results.

Question

TL;DR: Is there a clear explanation for why the memory-mapped file created by CreateFileMapping() in my DLL, which is loaded by Lync.exe, doesn't seem to be visible to my client application, causing OpenFileMapping() to fail with an error code of ERROR_FILE_NOT_FOUND?


Solution

  • As @Hans Passant pointed out, the referenced code made an unsafe typecast from char * to LPWSTR. By specifying my own Unicode strings explicitly and adjusting the constructors and internal variable types for osIPC::Client and osIPC::Server, I managed to get the IPC working.

    Depending on whether UNICODE was defined as a macro and some other project settings, this was leading to unpredictable behavior of the overall code.