Just as the title says, is it worth it going for a gitosis or gitolite setup instead of a simple git server setup?
Or, will it have any benefits in the future when I'll have more people pushing commits to that server?
Gitosis (or the more recent gitolite) makes it easy to use a git server for users not really used to linux and ssh and not having their own access.
If they're used to linux and they have to access the server for other reasons, and everybody has his own public repository on the server and everybody just pull from other repositories and push to his own ones, you don't really need gitosis. In this case, it's mainly an additional layer that you may avoid.