I can move over a word in Vim and hit Ctrl-w] and the tag opens in a horizontal split which is very handy, Ctrl-wf will do the same for files (very useful to open headers e.g.). Now how can I do the whole magic with vertical splits rather than horizontal?
Not sure if it exists. However, you can make your own binding:
nnoremap <C-W><C-V>f :exec "vert norm <C-V><C-W>f"<CR>
nnoremap <C-W><C-V>[ :exec "vert norm <C-V><C-W>["<CR>
This will bind Ctrl-WCtrl-Vf to be the vertical equivalent of Ctrl-Wf (and similar for tags).