This question and my answer below are mainly in response to an area of confusion in another question.
At the end of the answer, there are some issues WRT "volatile" and thread synchronisation that I'm not entirely confident about - I welcome comments and alternative answers. The point of the question primarily relates to CPU registers and how they are used, however.
CPU registers are small areas of data storage on the silicon of the CPU. For most architectures, they're the primary place all operations happen (data gets loaded in from memory, operated on, and pushed back out).
Whatever thread is running uses the registers and owns the instruction pointer (which says which instruction comes next). When the OS swaps in another thread, all of the CPU state, including the registers and the instruction pointer, get saved off somewhere, effectively freeze-drying the state of the thread for when it next comes back to life.
Lots more documentation on all of this, of course, all over the place. Wikipedia on registers. Wikipedia on context switching. for starters. Edit: or read Steve314's answer. :)