I'm attempting to simultaneously copy and write large amounts of data to on a server system using Windows Server 2012.
The writing happens from multiple applications with high data-rates, approaching approximately 30% of the I/O limits of the local system.
The transfer happens between the local storage and a network storage with a transfer rate of up to several gbit.
Because the applications writing the data must not suffer from low memory during their operation, the transfer must be aware of the load on the source system and possibly restrict itself during the transfer process.
Attempts have been made to apply basic Windows-based tools (x/copy, robocopy). x/copy is unsuitable because it does not supply throttling capabilites. Robocopy has proven either cause memory problems by it's throttling method or when not throttling by exceeding the available network/memory limits.
Now here's the fun part: Using the standard windows explorer copy GUI does not exhibit any of these problems. In fact, it copies blazingly fast while showing no noticeable impact at all on the system performance.
The next step would involve creating a custom copy process using basic win api to mimic the desirable explorer copy process behavior.
Q: What basic api copy command is the explorer copy using ... copyFile2? ... a completely unavailable internal copy command? Should I consider other options?
The IFileOperation interface lets you access the copy function Explorer uses.
The documentation for IFileOperation::CopyItem has a full example on how to use it.