I want to remove a specific watcher of my repository in Github. Is there a way to do that? Or can I even track who's watching my repository?
To my knowledge there is no way to remove watchers of your repository. When your repository is up on GitHub it becomes public (i.e., anyone can watch/fork it) assuming you have it as a public repository.
You can view who is watching your repository by clicking the number beside the watch/unwatch icon on the top-right of the repository page.
If for some reason it is crucial to remove a specific watcher (and you don't have that many watchers, or this would be a huge pain) you could delete the repository and then recreate it. This should remove any watchers, and start you fresh again. This is a huge pain for normal watchers though.