OSdev: deadlock using spinlock in keyboard interrupt handler

Question

I am writing an operating system in C++ and I set up a spin lock for the keyboard.

_Use_decl_annotations_
bool
HandleKeyboard(
    _In_ const BYTE Index,
    _In_ const PINTERRUPT_STACK Stack,
    _In_ const BYTE HasError,
    _In_ const PINTERRUPT_CPU_STATE Cpu
)
{
    (void)Index;
    (void)Stack;
    (void)HasError;
    (void)Cpu;

    if (const auto key = Pic::PicManager::keyboard.GetKeyPress(); key.hasValue)
    {
        __cli();
        {
            auto& [capitalize, keyLock, currentKey, wakeUp] = Pic::PicManager::keyboard;
            Sync::Guard guard(keyLock);

            currentKey = key.inner;
            wakeUp = true;
        }
        __sti();
    }


    return true;
}

It disables interrupts, acquires the lock and places the key into a global variable, then sets a flag so that the main loop can process it:

    while (true)
    {
        auto& [capitalize, keyLock, currentKey, wakeUp] = Pic::PicManager::keyboard;
        Sync::Guard guard(keyLock);

        if (!wakeUp || currentKey == 0)
        {
            continue;
        }
        
        // do something with the key
        
        currentKey = 0;
        wakeUp = false;
    }

Sometimes I can write 1-3 keys and then it gets stuck. Bochs says that the keyboard buffer is full, so I guess that it gets stuck in the interrupt handler somehow.

The implementation of spin lock:

__spinlock_lock:
    lock bts word [rcx], 0
    jc .wait
    ret
.wait:
    test word [rcx], 0
    jnz .wait
    jmp __spinlock_lock

__spinlock_unlock:
    lock btr word [rcx], 0
    ret

The argument in RCX is a volatile word. Sync::Spin::Lock just calls these functions, and Sync::Guard calls Lock in the constructor and Unlock in the destructor. I tried multiple implementations for the spin lock (basic while loop with a flag, <atomic>) but same thing. If I remove the lock from either one of the places it works.

I can't figure out how it gets stuck, since the while loop unlocks the spinlock at every iteration, so the interrupt spinlock has a chance to be acquired.

Answer 1

If the main loop has taken the lock, and the interrupt arrives on the same core during the critical section, then the handler will spin, but the lock can never become available because the main loop can't complete the critical section until after the handler returns. So yeah, of course it will deadlock.

Spinlocks are useful for mutual exclusion between threads running concurrently on different cores. You can't use them for mutual exclusion between a main thread and an interrupt handler on the same core, because they don't run concurrently; the main thread is suspended until the interrupt handler returns.

One approach here is to use a lock-free atomic flag for the handler to signal that data is available. When the main thread observes this flag to be set, it can then disable interrupts while retrieving data from the buffer.

Or, use a fully lock-free queue algorithm - preferably an off-the-shelf one because they are hard to get right.