I know Ctrl+Shift+Backspace is used to go to the location of the last edit.
But I want to jump to whichever location I was most recently at, not necessarily one where I edited anything.
For example, if I jumped to a declaration using Ctrl+B, then I want to go back to where I was before jumping.
Alt + Shift + ← (Left Arrow)
or
Ctrl + E (Recent Files pop-up).
Also check:
Ctrl + Shift + E (the Recently Edited Files pop-up).
Mac users, replace Ctrl with ⌘ (command) and Alt with ⌥ (option).
Update In v12.0 it's Alt + Shift +← (Left Arrow) instead of Alt + Ctrl + ← (Left Arrow).
Update 2 In v14.1 (and possibly earlier) it's Ctrl + [
Update 3 In IntelliJ IDEA 2016.3 it's Ctrl + Alt + ← (Left Arrow)
Update 4 In IntelliJ IDEA 2018.3 it's Alt + Shift + ← (Left Arrow)
Update 5 In IntelliJ IDEA 2019.3 it's Ctrl + Alt + ← (Left Arrow)