Debugging Memory Mapped Files

Question

I’m developing a .NET program which suppose to be communicate with other existing program in same machine using Shared Memory (Memory Mapped Files), The existing program itself include a native dll and .NET wrapper / bridge dll which I could consume to communicate with the other program.

The thing is the other program is very buggy and the source codes of the program and dll are long gone, so my goal is to recreate the other program. I could decompile .NET dll but it just straight P/Invoke the native dll.

Fortunately, I able to view the memory mapped file with System.IO.MemoryMappedFiles.MemoryMappedFile but that’s all I got, I have no idea in which offset the dll read the data from memory mapped file when I call certain functions from it (Most of the time the dll only doing read and no write)

I also tried to hook CreateFileMapping, OpenFileMapping and MapViewOfFile in C++ program and call certain functions inside the native dll but those hooked function only called once in during initialization and did not trigger when invoke function in dll that suppose to read data from memory mapped file. If I dare to guess, the dll must be has pointer to the memory mapped file.

So my question, is it possible to put breakpoint or detect when any process attempt to read / write of certain region of memory mapped file? For an instance, I'd like to know whether there's read process or pointer pointing at offset 0xdeadbeef in my memory mapped file. I'm open to alternative, so let me know if you had any better idea

Answer 1

I'm not familiar with Memory Mapped File but if the two process share same memory, I believe it has same physical address, if this is true then you could probably use Cheat Engine with kernel mode.

Just keep in mind that it can cause system instability or crash so make sure to save all your works before proceeding. You can follow this steps:

1. First open one of the program, make sure that "shared memory" is created before proceeding into the next step.
2. Open Cheat Engine, make sure to enable kernel mode for OpenProcess in Settings -> Extra and enable kernel debugger in Settings -> Debugger Options (otherwise you won't be able to observe the instructions that access the memory)
3. Open the process in Cheat Engine, select the program that you launched in the first step
4. Now scan the memory inside cheat engine, since you mentioned that you able to view the memory, take out some chunk of bytes that you wish to monitor, change value type to array of byte and scan for those bytes
5. Scan result will appear in the left pane, you might found multiple result, there's only one correct address most of the time and the rest is accidentally match with the value you scan for, right click and select Browse this memory region, make sure that surrounding data match with what you expect
6. Back to main window, double click the correct ones and it will appear on the bottom pane, right click it and select "DBVM Find out what writes or access this address", a dialog will appear (or your system might crash), the most important part is ensure that the physical address there instead appearing at textbox
7. Select access type, you could watch only writes or both read and writes and hit "Start watch" button
8. Every time any process make an attempt to access or write into that memory region, new instruction will appear in the list.

In my opinion, this topic is close to reverse engineering. Since you're open to suggestion, here's my other two cents: if you had a plan to re-create the existing program which emit the data via shared memory and you're developing the client too, why not remake both of them from scratch so you don't have to deal with this?