Story: I am developing a KMDF driver for a custom PCI-E board, which has a PLX9056 chip connecting to the PCI-E.
The driver and it's test software works perfectly after booting the computer, but after restarting the software and driver a several times, the transfer speed slows down from 160Mb/s to around 110 MB/s step by step.
(I have to disable and re-enable the driver everytime I want to restart the software, firmware bug, not my job atm.)
My main question is, is there any memory or anything connected to the DMA that can get fragmented and therefore make the transfers slower?
The driver uses Scatter/Gather DMA, hardware supported, only 32bit. This only occurs when transferring data to the device.
The issue seems to be the way the driver handles the data/buffers. The PLX provided DMA testing software always gets the 160MB/s rate every time, even when our driver already broke and manages only 110MB/s. Therefore, the problem must be in our driver.
Reinstalling the driver does not solve the problem, only a reboot does.
Worth to mention, the software allocates a rather huge buffer (1GB), from which it sends the data to the device through the driver.
Solved the issue.
The fragmentation of the memory caused the driver to not be able to allocate large continous blocks of memory after the OS has been running for a while.
Fixed by using Large-Page Support which allowed me to allocated huge continous memory pages. https://learn.microsoft.com/en-us/windows/win32/memory/large-page-support