I use VisualSVN as svnserver which manages the repository stored in X: drive. Not everyone has access to X: drive, but i want to give some of them access to repository. Do they need to have access to X: drive or passing authorization of visualsvn is enough?
The user account under which VisualSVN Server is running must have access to the location. Your other users should not have write permission there; you risk repository data loss/corruption if you do. All access, authorization & authentication to the repository should be managed through the Subversion server, not file/share/drive permissions.
Do not depend upon mapped drives for Windows services. If you must use a network share/path, use the UNC path. Drive letters are only usable for interactive user sessions.
If at all possible, keep your repository on a drive that is local to your Subversion server process, not a network share. More recent versions have mitigated the risks, but it's still recommended that you be on a local path.