Say I have a site on http://example.com. I would really like allowing bots to see the home page, but any other page need to blocked as it is pointless to spider. In other words
http://example.com & http://example.com/ should be allowed, but http://example.com/anything and http://example.com/someendpoint.aspx should be blocked.
Further it would be great if I can allow certain query strings to passthrough to the home page: http://example.com?okparam=true
So after some research, here is what I found - a solution acceptable by the major search providers: google , yahoo & msn (I could on find a validator here) :
User-Agent: *
Disallow: /*
Allow: /?okparam=
Allow: /$
The trick is using the $ to mark the end of URL.