Search code examples
iopcipci-ebase-address

How to calculate size of MMIO-mapped region from BAR address in PCIe

I've been diving deeper into how PCIe works in general, and I'm stuck at where many books and websites talk about PCIe configuration space.
What I have learned so far is that for each of the assigned device with its BDF (bus-device-function bits), there corresponds a 4KB configuration space for that device, which includes the 64B region as below:
64B space
I understand up to the point where each of the base address registers (that represent memory-mapped region) are decoded as below:
bar decoding
(both of the images are from this site)

What I don't get is how the size of the region is determined. For example, in one server, I get the following for a GPU connected to BDF 00:05.0 command lspci -x -v -s 05:00.0:

05:00.0 VGA compatible controller: NVIDIA Corporation GV100 [TITAN V] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation GV100 [TITAN V]
    Flags: bus master, fast devsel, latency 0, IRQ 80, NUMA node 0
    Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
    Memory at a0000000 (64-bit, prefetchable) [size=256M]
    Memory at b0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at d000 [size=128]
    [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
00: de 10 81 1d 07 05 10 00 a1 00 00 03 00 00 80 00
10: 00 00 00 f8 0c 00 00 a0 00 00 00 00 0c 00 00 b0
20: 00 00 00 00 01 d0 00 00 00 00 00 00 de 10 18 12
30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00

We can see that the value of BAR0 is 0xf8000000. But how do we know the size of the region that starts at address 0xf8000000? From some sites I've checked (one, two, and three) they talk about:
(1) finding the complement value of the address which should be the length of the region (which does not make sense to me in a few ways) or
(2) since 0xf8000000 is 1111 1000 0000 0000 0000 0000 0000 0000, the size of the region is 2^27=128MB because there are 27 consecutive 0s until it meets the first 1.

But both of these methods are wrong, because the lspci command said that particular region has 16MB mapped, not 128MB.

So here are my real questions: 1. How exactly should the memory region size be calculated? 2. Also, the memory mapped above seems to add up to 16M+256M+32M+128(+128K), but the actual size of the GPU memory is a little over 12GB. Is it right that not all of the GPU memory is mapped with MMIO via PCIe?

Thanks in advance.

Solution

  • The OSDev link you mentioned defines the protocol for probing the BAR:

    To determine the amount of address space needed by a PCI device, you must save the original value of the BAR, write a value of all 1's to the register, then read it back. The amount of memory can then be determined by masking the information bits, performing a bitwise NOT ('~' in C), and incrementing the value by 1. The original value of the BAR should then be restored. The BAR register is naturally aligned and as such you can only modify the bits that are set.