It seems to make no difference for the repo I'm working with, and the man page is not very helpful.
git fetch --all
makes a difference if you have multiple remotes. Most of the time you have only "origin", but you can add any number of remotes.
git clone ssh://example.com/git/repo/first-repo
git remote add my-other-remote ssh://example.com/git/repo/another
git fetch
will fetch from "first-repo", where git fetch --all
will fetch from "first-repo" and "another"