Currently I have only figured out how I can use TfsTeamProjectCollection and VersionControlServer to get all files related to a specific changeset (i.e. added/changed/deleted files etc).
What I can't figure out how to get all files like the "Get specific version" in the source code explorer.
How do I do that without going through all change sets up to the wanted one?
Use the VersionControlServer.GetItems method. Before you can call this, you need to define a WorkSpace and map the paths you're interested in to your local disk.
The GetItems method has a number of overloads that accept a VersionSpec. You can use VersionSpec.Latest to get the latest version, but you can also create a VersionSpec based on a label name, date/time or changeset number.
GetItems will always get the last change that happened on or before that spec, so there is no need to call GetItems for all changesets up to a specific one.