I have a PCIe model written in System Verilog, although I think this question is language agnostic. The model performs PCIe configuration reads and writes and memory reads and writes perfectly in simulation. However, what I need to do is "discover" my PCIe device and configure my config space registers in simulation. Is there a boiler plate chunk of pseudo code that represents the Linux PCIe enumeration process that I can just add my own models transactions functions too so that I can get a "Bus walk", followed by BAR programming, SR-IOV enable if discovered, MSIx config? It seems like this would be a common exercise for PCIe device so maybe there is model.
It isn't terribly difficult to do. Basically you loop through the config space, checking for each each possible device on the first root bus 0. When a device is found, you allocate a memory space for it based on its requested size and program the BARs accordingly. If you find any bridges, you also configure and enable them - the basic bridge registers for this are standard. This includes assigning the upstream and downstream bus numbers, which then allows you to enumerate the new downstream bus, and so on.
I had to do this once to access a PCI I/O card on a system that had no OS or other software environment. It wasn't too bad and that was across two bridges from two vendors, as well as the I/O card registers and the CPU bus root bridge setup. This was PCI, not PCIe, but it would be very much the same. You could even do it with completely hard-coded numbers if the hardware never changed, but in my case there were a couple variants so I actually had to do some simple enumeration to find the device numbers dynamically. One gotcha is that you may have to delay a bit, or retry, to give all the devices time to come online before you try to access them.
In doing that I found this book to be invaluable: PCI System Architecture (4th Edition). I notice there is also an version for PCIe: PCI Express System Architecture (1st Edition). I would definitely get one of those if you haven't already. These books contain detailed algorithms and explanations about how to do all of this. At the time I didn't really use or refer to any code to speak of, but...
The best code resource I have found is U-Boot. It operates at a similarly low-level and is totally self contained and is still fairly small and as simple as possible. For example, the enumeration appears to start with the function pci_init()
calls a board specific pci_xxx_init()
. This then sets up the root bridge and then calls pci_hose_scan_bus()
in drivers/pci/pci.c to do the real work. Also check out the routines in drivers/pci/pci_auto.c, as well as the rest of the folder.
For your task you probably only need a very small subset and could just hack out parts of these files into a simple driver. Basically a for() loop and some pci_read/write_config() calls with logic to recognize your device and bridge IDs.