I'm developing a microkernel for my personal research. I have chosen to run my kernel at 0xf0000000, leaving 3.75 GiB for user space programs. When my kernel starts up, it sets up 32-bit paging (with hardcoded page directory and page tables). Then it checks if PAE is supported on host machine and sets up page directory pointer table (PDPT). But the problem comes when I try to load it into %cr3. According to the Intel Software Developer Manual:
Software can transition between 32-bit paging and PAE paging by changing the value of CR4.PAE with MOV to CR4.
So tried to use the following code to switch to PAE paging:
movl %cr4, %eax
orl $(1 << 5), %eax
movl %eax, %cr4
movl %ebx, %cr3 // %ebx holds physical address to PDPT
Or, in Intel syntax (or NASM):
mov eax, cr4
or eax, 1 << 5
mov cr4, eax
mov cr3, ebx // ebx holds physical address to PDPT
But it fails (on QEMU). It writes to %cr4, sets %eip to next instruction, executes it (atleast GDB says this), and resets. I tried to write to %cr3 before %cr4, but still the same result.
Then I tried to switch to PAE paging by: unset PG -> set PAE -> write to %cr3 -> set PG and I succeeded. But I want switch to PAE paging directly. How is that possible?
Then I tried to switch to PAE paging by: unset PG -> set PAE -> write to %cr3 -> set PG and I succeeded. But I want switch to PAE paging directly. How is that possible?
It's not possible.
If "plain paging" is already in use/enabled, then you can't atomically enable PAE and load CR3 at the same time, so (regardless of whether you load CR3 first then CR4, or load CR4 first then try to load CR3) whichever instruction happens first will make the CPU crash before the second instruction is fetched.
The only way is to temporarily disable paging.