Git stash knows what branch you were working on, e.g.:
stash@{1}: WIP on master: c264051 Revert "added file_size"
Is it possible to have git stack pop switch back to that branch before popping, or do I have to do it manually?
No, since stash entries don't record any information about branches by default.
You'll have to do it manually, and the reference to the branch (WIP on <branch> in the stash message) is only informative : you could as well pop that stash anywhere else, as it's not technically linked to the branch it was created from.
If you create your stash with git stash save and a custom message where you don't include the branch name, you'll have no way of knowing where it was from (not that it's always needed, btw).
Also, interestingly, you could stash from no branch at all, with a detached HEAD state, the stash automatic message would contain
WIP on (no branch): abc123 <text (subject) of commit abc123's message>