How does Windows switch to supervisor mode during a system call? I heard something about a "trap 0", but that doesn't even seem like an x86 instruction. I stepped through some system calls, but I can't find any. Do a lot of Windows system calls run in user mode? Which DO run in supervisor mode?
A system call is also known as a software interrupt. The x86 instruction which calls a software interrupt has the mnemonic INT. How data is passed to the operative system is defined by the operative system ABI. As far as I know, Windows uses the immediate 0x80 for all its routines and sends additional data via registers, but I'm not sure. 0x20 is the first available immediate, since the range 0 through 31 is reserved and used for general exceptions like integer division by zero and memory faults.
What basically happens is that the CPU changes to privileged mode and reads the IDTR (Interupt Descriptor Table Register). There it finds the physical memory address for the IDT (Interupt Descriptor Table) and does a lookup into the IDT, based on the 8-bit immediate baked into the software interrupt instruction. The IDT can be stored wherever in memory. The IDTR can be read/written by the instructions LIDT and SIDT. The IDT can store a variety of information, but for interrupts it stores the address to the service routine associated with the INT immediate.
Examples of win32-functions which fires a software interrupt.. hm. printf and friends sure does, as does EnterCriticalSection. In Windows Vista and Windows 7, some OpenGL and DirectX API calls now require a roundtrip into kernel land due to the new composite manager. For OpenGL, that applies for all functions who reads the current backbuffer, like glReadPixels, glCopy(Sub)TexImage2D, etc.
P.S: Take this post with a pinch of salt. It's been a while since I messed around with Windows this way, and I didn't perform a lot of fact checking. Edits and comments are welcome.
And here is a link to the original Intel 386 manual (which I quoted anyway)