How can I deny access to http://sub.mydomain.com/, but allow for (completely) http://sub.mydomain.com/test (or http://sub.mydomain.com/test/)
There is a magento back-end behind http://sub.mydomain.com/test/
.htaccess directives apply to that directory, and all subdirectories thereof, so you should disallow access in your DocumentRoot,
http://sub.mydomain.com/.htaccess:
Order deny,allow
Deny from all
And override that in any specific subdirectories you would like to allow access to,
http://sub.mydomain.com/test/.htaccess:
Order allow,deny
Allow from all