I'm reading about interrupt handling in mondern CPUs and operating systems, but I can't figure out one point:
As soon as some hardware device changes the state (current/voltage?) on an interrupt pin of the CPU, the CPU stops after processing the prevailing instruction and jumps to execute the interrupt handler code. Now imagine the interrupt handler code has to change some kind of state in scheduler's data structures, however before the OS was interrupted it was also fumbling around in the same structures. That would lead to messed up data, so there must be a solution.
I would guess the OS and the interrupt handler both use a semaphore, implemented through some atomic compare/set memory operation to protect the shared data structures. However, if the OS gets interrupted while holding such a semaphore, the interrupt handler could not do anything and the interrupt would just vanish, because busy waiting for that semaphore would never return control to the OS, hence the lock is never released.
How is this problem solved? There must be some trick that I'm missing...
Maybe an hardware detail you are missing can explain your confusion.
Whenever an hardware interrupt occurs, something along these lines happen:
1 - The CPU goes to a privileged mode, further hardware interrupts are disabled (normally a bit in the processor flags register), and execution jumps to the interrupt handler.
2 - Once the OS interrupt handling is done, it re-enables hardware interrupts, so further interrupts can happen.
So, in short, the OS/interrupt handler can control when hardware interrupts are allowed to interrupt the normal flow. An easy solution to your problem would be just have the OS disable hardware interrupts while messing with those data structures. In practice, things get more complex to minimize interrupt latency.
Things can change from one architecture to another, but the basic principle is still that further hardware interrupts are disabled when one happens, and they can be enabled/disabled (provided the CPU is running in the required privileged modes).
Check the end part of this: http://en.wikibooks.org/wiki/X86_Assembly/Advanced_Interrupts