Search code examples
asp.netvb.netindexingrobots.txtdatapager

is it possible to stop search engines indexing pages with certain querystring parameter

I am using the asp.net DataPager and noticed that search engines wouldn't be able to crawl the links created, so I added a querystring parameter and now see that it could crawl the links.

The problem I now have is that it will now eventially end up with multiple urls for the same page.

Is it possible to stop search engines indexing all pages with the query string parameter such as:

?pg=

so the search engine will follow these links but not index:

http://domain.co.uk/news/07-04-2011/bank-of-england-base-rate?pg=3
http://domain.co.uk/news/11-03-2011/client-feedback-from-employee-relocation?pg=6

but would index:

http://domain.co.uk/news/07-04-2011/bank-of-england-base-rate
http://domain.co.uk/news/11-03-2011/client-feedback-from-employee-relocation

Thanks for any help

J.

Solution

  • I'm pretty sure you can achieve that with a robots.txt

    With google you can specify parameters you want to exclude with the webmaster tools.

    You could also trap the useragent and simply crash the page when it's a bot ...