Search code examples
kernelx86-64pagingosdevpage-tables

Why are my page tables not mapping the right physical addresses?

I am writing a small x86-64 kernel booted with UEFI. I think there is something I overlook in the code and I can't figure out what it is. I am trying to page the xHCI BAR in my higher half kernel. I've got this small code snippet which is supposed to do that:

__attribute__((aligned (4096))) UINT64 Paging::xhciPageDir;
__attribute__((packed)) __attribute__((aligned (4096))) UINT64 Paging::xhciPageTable[512];

void Paging::PageXHCIConfigSpace(UINT64 baseAddrRegSpace, UINT64 barAreaSize){
    /*
    Divide the BAR address by 1GB (0x40000000) to determine the position in the PDP
    */
    UINT64 pdpEntryNum = baseAddrRegSpace / 0x40000000;
    
    UINT64* pdpPtr = (UINT64*)0xffff800000601000; //Pointer to the PDP
    /*
    Make the PDP entry point to the page directory for the xHCI ored with 0x1b to mark
    the page as present and uncached minus 0xffff800000000000 because the PDP contains physical
    addresses.
    */
    *(pdpPtr + pdpEntryNum) = (((UINT64)&xhciPageDir) | 0x1b) - 0xffff800000000000;
    
    //Make the first entry of the PD of the xHCI point to the page table.
    xhciPageDir = (((UINT64)&xhciPageTable) | 0x1b) - 0xffff800000000000;
    
    /*
    Make the page table 4th entry map to the BAR address
    */
    UINT64 ptEntry = baseAddrRegSpace | 0x1b;
    for (UINT32 i = 4; i < 4 + (barAreaSize / 4096); i++){
        xhciPageTable[i] = ptEntry;
        ptEntry += 0x1000;
    }
}

This code snippet is called by this other code snippet:

    /*Bar0 and Bar1 combining*/
    UINT32* pciPtr = (UINT32*)configSpacePhysAddr;
    UINT32 register4 = *(pciPtr + 4);
    UINT32 register5 = *(pciPtr + 5);
    UINT64 bar0 = (UINT64)register4;
    UINT64 bar1 = (UINT64)register5;
    baseAddrRegSpace = (bar1 << 32) + (bar0 & 0xfffffff0);
    
    *(pciPtr + 4) = 0xffffffff;
    UINT32 barAreaSize = *(pciPtr + 4);
    barAreaSize &= 0xfffffff0;
    barAreaSize = ~barAreaSize;
    barAreaSize += 1;
    Paging::PageXHCIConfigSpace(baseAddrRegSpace, barAreaSize);

The first part of this code snippet gets the BAR address by combining BAR1 and BAR0 together. The address pointed to by the BARs is 0x800004000. It is 4 pages past the 32nd GB. Then I use the second part of this code snippet to calculate the size of the BAR area pointed to by the BARs. It comes out to being 4 pages or 16384 bytes in size.

In the first code snippet I thus make some assumptions that I need to fill the 4th entry of the page table to map 0xffff800800004000 to 0xffff800800008000. I then fill these 4 page table entries with 0x800004000, 0x800005000, 0x800006000 and 0x800007000. This seems to work properly but when I use monitor info mem in GDB it returns the following:

(gdb) monitor info mem
ffff800000000000-ffff800100000000 0000000100000000 -rw
ffff800800004000-ffff800800008000 0000000000004000 -rw

I have the right virtual addresses but they are mapped from 0x0000 to 0x4000 which isn't the right physical addresses. I thus looked in memory to find out the culprit. Here's several memory dumps which show all the page tables:

(gdb) dump memory result.bin 0xffff800000601000 0xffff800000602000
(gdb) dump memory result.bin 0xffff800000511000 0xffff800000512000
(gdb) dump memory result.bin 0xffff800000512000 0xffff800000513000

user@user-System-Product-Name:~$ hexdump -C result.bin
00000000  3b 20 60 00 00 00 00 00  1b 30 60 00 00 00 00 00  |; `......0`.....|
00000010  1b 40 60 00 00 00 00 00  3b 50 60 00 00 00 00 00  |.@`.....;P`.....|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  1b 10 51 00 00 00 00 00  00 00 00 00 00 00 00 00  |..Q.............|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000
user@user-System-Product-Name:~$ hexdump -C result.bin
00000000  1b 20 51 00 00 00 00 00  00 00 00 00 00 00 00 00  |. Q.............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000
user@user-System-Product-Name:~$ hexdump -C result.bin
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000020  1b 40 00 00 08 00 00 00  1b 50 00 00 08 00 00 00  |[email protected]......|
00000030  1b 60 00 00 08 00 00 00  1b 70 00 00 08 00 00 00  |.`.......p......|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

In the page tables (in the last dump) it shows the right physical addresses. They are not reflected in the monitor info mem command with GDB. After some time looking around I still don't understand why.

Anybody got any idea of what is wrong?

EDIT

Here's the PML4 output:

(gdb) dump memory result.bin 0xffff800000600000 0xffff800000601000

user@user-System-Product-Name:~$ hexdump -C result.bin
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800  3b 10 60 00 00 00 00 00  00 00 00 00 00 00 00 00  |;.`.............|
00000810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

EDIT 2

In the end I used monitor info tlb to have more details. I didn't use it before because the first 4GB mapping causes a lot of output. The end of the output looks like such:

ffff8000ffff9000: 00000000ffff9000 -----CT-W
ffff8000ffffa000: 00000000ffffa000 -----CT-W
ffff8000ffffb000: 00000000ffffb000 -----CT-W
ffff8000ffffc000: 00000000ffffc000 -----CT-W
ffff8000ffffd000: 00000000ffffd000 -----CT-W
ffff8000ffffe000: 00000000ffffe000 -----CT-W
ffff8000fffff000: 00000000fffff000 -----CT-W
ffff800800004000: 0000000800004000 -----CT-W
ffff800800005000: 0000000800005000 -----CT-W
ffff800800006000: 0000000800006000 -----CT-W
ffff800800007000: 0000000800007000 -----CT-W

According to monitor info tlb, my mapping is correct. Can this be cache related? It is weird because I would think that the wrong mapping should be in monitor info mem not in tlb. If the right mapping is in the TLB then it should work properly. Is there some kind of bug in monitor info mem?

EDIT 3

I am still working on it and I got a physical memory mapping from the monitor xp/fmt command. Here it goes:

(gdb) monitor xp/260xg 0x600000
0000000000600000: 0x0000000000000000 0x0000000000000000
0000000000600010: 0x0000000000000000 0x0000000000000000
0000000000600020: 0x0000000000000000 0x0000000000000000
0000000000600030: 0x0000000000000000 0x0000000000000000
0000000000600040: 0x0000000000000000 0x0000000000000000
0000000000600050: 0x0000000000000000 0x0000000000000000
0000000000600060: 0x0000000000000000 0x0000000000000000
0000000000600070: 0x0000000000000000 0x0000000000000000
0000000000600080: 0x0000000000000000 0x0000000000000000
0000000000600090: 0x0000000000000000 0x0000000000000000
00000000006000a0: 0x0000000000000000 0x0000000000000000
00000000006000b0: 0x0000000000000000 0x0000000000000000
00000000006000c0: 0x0000000000000000 0x0000000000000000
00000000006000d0: 0x0000000000000000 0x0000000000000000
00000000006000e0: 0x0000000000000000 0x0000000000000000
00000000006000f0: 0x0000000000000000 0x0000000000000000
0000000000600100: 0x0000000000000000 0x0000000000000000
0000000000600110: 0x0000000000000000 0x0000000000000000
0000000000600120: 0x0000000000000000 0x0000000000000000
0000000000600130: 0x0000000000000000 0x0000000000000000
0000000000600140: 0x0000000000000000 0x0000000000000000
0000000000600150: 0x0000000000000000 0x0000000000000000
0000000000600160: 0x0000000000000000 0x0000000000000000
0000000000600170: 0x0000000000000000 0x0000000000000000
0000000000600180: 0x0000000000000000 0x0000000000000000
0000000000600190: 0x0000000000000000 0x0000000000000000
00000000006001a0: 0x0000000000000000 0x0000000000000000
00000000006001b0: 0x0000000000000000 0x0000000000000000
00000000006001c0: 0x0000000000000000 0x0000000000000000
00000000006001d0: 0x0000000000000000 0x0000000000000000
00000000006001e0: 0x0000000000000000 0x0000000000000000
00000000006001f0: 0x0000000000000000 0x0000000000000000
0000000000600200: 0x0000000000000000 0x0000000000000000
0000000000600210: 0x0000000000000000 0x0000000000000000
0000000000600220: 0x0000000000000000 0x0000000000000000
0000000000600230: 0x0000000000000000 0x0000000000000000
0000000000600240: 0x0000000000000000 0x0000000000000000
0000000000600250: 0x0000000000000000 0x0000000000000000
0000000000600260: 0x0000000000000000 0x0000000000000000
0000000000600270: 0x0000000000000000 0x0000000000000000
0000000000600280: 0x0000000000000000 0x0000000000000000
0000000000600290: 0x0000000000000000 0x0000000000000000
00000000006002a0: 0x0000000000000000 0x0000000000000000
00000000006002b0: 0x0000000000000000 0x0000000000000000
00000000006002c0: 0x0000000000000000 0x0000000000000000
00000000006002d0: 0x0000000000000000 0x0000000000000000
00000000006002e0: 0x0000000000000000 0x0000000000000000
00000000006002f0: 0x0000000000000000 0x0000000000000000
0000000000600300: 0x0000000000000000 0x0000000000000000
0000000000600310: 0x0000000000000000 0x0000000000000000
0000000000600320: 0x0000000000000000 0x0000000000000000
0000000000600330: 0x0000000000000000 0x0000000000000000
0000000000600340: 0x0000000000000000 0x0000000000000000
0000000000600350: 0x0000000000000000 0x0000000000000000
0000000000600360: 0x0000000000000000 0x0000000000000000
0000000000600370: 0x0000000000000000 0x0000000000000000
0000000000600380: 0x0000000000000000 0x0000000000000000
0000000000600390: 0x0000000000000000 0x0000000000000000
00000000006003a0: 0x0000000000000000 0x0000000000000000
00000000006003b0: 0x0000000000000000 0x0000000000000000
00000000006003c0: 0x0000000000000000 0x0000000000000000
00000000006003d0: 0x0000000000000000 0x0000000000000000
00000000006003e0: 0x0000000000000000 0x0000000000000000
00000000006003f0: 0x0000000000000000 0x0000000000000000
0000000000600400: 0x0000000000000000 0x0000000000000000
0000000000600410: 0x0000000000000000 0x0000000000000000
0000000000600420: 0x0000000000000000 0x0000000000000000
0000000000600430: 0x0000000000000000 0x0000000000000000
0000000000600440: 0x0000000000000000 0x0000000000000000
0000000000600450: 0x0000000000000000 0x0000000000000000
0000000000600460: 0x0000000000000000 0x0000000000000000
0000000000600470: 0x0000000000000000 0x0000000000000000
0000000000600480: 0x0000000000000000 0x0000000000000000
0000000000600490: 0x0000000000000000 0x0000000000000000
00000000006004a0: 0x0000000000000000 0x0000000000000000
00000000006004b0: 0x0000000000000000 0x0000000000000000
00000000006004c0: 0x0000000000000000 0x0000000000000000
00000000006004d0: 0x0000000000000000 0x0000000000000000
00000000006004e0: 0x0000000000000000 0x0000000000000000
00000000006004f0: 0x0000000000000000 0x0000000000000000
0000000000600500: 0x0000000000000000 0x0000000000000000
0000000000600510: 0x0000000000000000 0x0000000000000000
0000000000600520: 0x0000000000000000 0x0000000000000000
0000000000600530: 0x0000000000000000 0x0000000000000000
0000000000600540: 0x0000000000000000 0x0000000000000000
0000000000600550: 0x0000000000000000 0x0000000000000000
0000000000600560: 0x0000000000000000 0x0000000000000000
0000000000600570: 0x0000000000000000 0x0000000000000000
0000000000600580: 0x0000000000000000 0x0000000000000000
0000000000600590: 0x0000000000000000 0x0000000000000000
00000000006005a0: 0x0000000000000000 0x0000000000000000
00000000006005b0: 0x0000000000000000 0x0000000000000000
00000000006005c0: 0x0000000000000000 0x0000000000000000
00000000006005d0: 0x0000000000000000 0x0000000000000000
00000000006005e0: 0x0000000000000000 0x0000000000000000
00000000006005f0: 0x0000000000000000 0x0000000000000000
0000000000600600: 0x0000000000000000 0x0000000000000000
0000000000600610: 0x0000000000000000 0x0000000000000000
0000000000600620: 0x0000000000000000 0x0000000000000000
0000000000600630: 0x0000000000000000 0x0000000000000000
0000000000600640: 0x0000000000000000 0x0000000000000000
0000000000600650: 0x0000000000000000 0x0000000000000000
0000000000600660: 0x0000000000000000 0x0000000000000000
0000000000600670: 0x0000000000000000 0x0000000000000000
0000000000600680: 0x0000000000000000 0x0000000000000000
0000000000600690: 0x0000000000000000 0x0000000000000000
00000000006006a0: 0x0000000000000000 0x0000000000000000
00000000006006b0: 0x0000000000000000 0x0000000000000000
00000000006006c0: 0x0000000000000000 0x0000000000000000
00000000006006d0: 0x0000000000000000 0x0000000000000000
00000000006006e0: 0x0000000000000000 0x0000000000000000
00000000006006f0: 0x0000000000000000 0x0000000000000000
0000000000600700: 0x0000000000000000 0x0000000000000000
0000000000600710: 0x0000000000000000 0x0000000000000000
0000000000600720: 0x0000000000000000 0x0000000000000000
0000000000600730: 0x0000000000000000 0x0000000000000000
0000000000600740: 0x0000000000000000 0x0000000000000000
0000000000600750: 0x0000000000000000 0x0000000000000000
0000000000600760: 0x0000000000000000 0x0000000000000000
0000000000600770: 0x0000000000000000 0x0000000000000000
0000000000600780: 0x0000000000000000 0x0000000000000000
0000000000600790: 0x0000000000000000 0x0000000000000000
00000000006007a0: 0x0000000000000000 0x0000000000000000
00000000006007b0: 0x0000000000000000 0x0000000000000000
00000000006007c0: 0x0000000000000000 0x0000000000000000
00000000006007d0: 0x0000000000000000 0x0000000000000000
00000000006007e0: 0x0000000000000000 0x0000000000000000
00000000006007f0: 0x0000000000000000 0x0000000000000000
0000000000600800: 0x000000000060103b 0x0000000000000000
0000000000600810: 0x0000000000000000 0x0000000000000000
(gdb) monitor xp/64xg 0x601000
0000000000601000: 0x000000000060203b 0x000000000060301b
0000000000601010: 0x000000000060401b 0x000000000060503b
0000000000601020: 0x0000000000000000 0x0000000000000000
0000000000601030: 0x0000000000000000 0x0000000000000000
0000000000601040: 0x0000000000000000 0x0000000000000000
0000000000601050: 0x0000000000000000 0x0000000000000000
0000000000601060: 0x0000000000000000 0x0000000000000000
0000000000601070: 0x0000000000000000 0x0000000000000000
0000000000601080: 0x0000000000000000 0x0000000000000000
0000000000601090: 0x0000000000000000 0x0000000000000000
00000000006010a0: 0x0000000000000000 0x0000000000000000
00000000006010b0: 0x0000000000000000 0x0000000000000000
00000000006010c0: 0x0000000000000000 0x0000000000000000
00000000006010d0: 0x0000000000000000 0x0000000000000000
00000000006010e0: 0x0000000000000000 0x0000000000000000
00000000006010f0: 0x0000000000000000 0x0000000000000000
0000000000601100: 0x000000000051101b 0x0000000000000000
0000000000601110: 0x0000000000000000 0x0000000000000000
0000000000601120: 0x0000000000000000 0x0000000000000000
0000000000601130: 0x0000000000000000 0x0000000000000000
0000000000601140: 0x0000000000000000 0x0000000000000000
0000000000601150: 0x0000000000000000 0x0000000000000000
0000000000601160: 0x0000000000000000 0x0000000000000000
0000000000601170: 0x0000000000000000 0x0000000000000000
0000000000601180: 0x0000000000000000 0x0000000000000000
0000000000601190: 0x0000000000000000 0x0000000000000000
00000000006011a0: 0x0000000000000000 0x0000000000000000
00000000006011b0: 0x0000000000000000 0x0000000000000000
00000000006011c0: 0x0000000000000000 0x0000000000000000
00000000006011d0: 0x0000000000000000 0x0000000000000000
00000000006011e0: 0x0000000000000000 0x0000000000000000
00000000006011f0: 0x0000000000000000 0x0000000000000000
(gdb) monitor xp/16xg 0x511000
0000000000511000: 0x000000000051201b 0x0000000000000000
0000000000511010: 0x0000000000000000 0x0000000000000000
0000000000511020: 0x0000000000000000 0x0000000000000000
0000000000511030: 0x0000000000000000 0x0000000000000000
0000000000511040: 0x0000000000000000 0x0000000000000000
0000000000511050: 0x0000000000000000 0x0000000000000000
0000000000511060: 0x0000000000000000 0x0000000000000000
0000000000511070: 0x0000000000000000 0x0000000000000000
(gdb) monitor xp/16xg 0x512000
0000000000512000: 0x0000000000000000 0x0000000000000000
0000000000512010: 0x0000000000000000 0x0000000000000000
0000000000512020: 0x000000080000401b 0x000000080000501b
0000000000512030: 0x000000080000601b 0x000000080000701b
0000000000512040: 0x0000000000000000 0x0000000000000000
0000000000512050: 0x0000000000000000 0x0000000000000000
0000000000512060: 0x0000000000000000 0x0000000000000000
0000000000512070: 0x0000000000000000 0x0000000000000000

The mapping seems right. I think the problem is caching but the CR3 register is 0x600018 (cache disabled bit set). Caching should basically be disabled.

Solution

  • Basically, I used this link to find out the size of the config space pointed to by the BAR: How is a PCI / PCIe BAR size determined?.

    This is misleading because it doesn't also state that, when you do this, the BAR changes position and it broke my code.

    It is normal that the monitor info mem command returns the wrong addresses because the memory pointed to by the BAR (above the 32nd GB) doesn't exist. The command thus wraps back to 0x4000 instead of returning the right addresses. In the meantime, monitor info tlb returns the right addresses. This is probably just software behavior. I found out a link which states that info tlb name is misleading because it doesn't actually return the TLB entries. It simply reads the page tables and returns what it finds.

    In the end, everything is fixed. I simply have to rewrite the xHCI BAR with its last value to make the code work. Even then, the monitor info mem command doesn't return the right addresses but the mapping actually used maps to the right addresses. My confusion came from the fact that reading the registers of the xHC always returned 0. When I used info mem, it returned the wrong addresses so I thought this was the culprit while the actual culprit was with the code to check the size of the config space pointed to by the BAR.