I have created a web app in .net for an alcohol company.
The client required that all access to the website be denied unless the user agrees they are of age in their county and they agree to the terms.
To implement this functionality I simply redirect all traffic to a verification page where they can set a new session to say they have agreed (on this visit) to the terms. once the session is set, the site becomes fully accessible.
My problem is, I would like google et al to crawl and index all the pages on the site. How can I make sure the bots don't get caught in the age verification page, what is the best practice here to ensure my whole site gets crawled.
The HttpWebRequest of the .NET Framework takes care of most of that.
In the Page_Load of your code behind file you can easily check for crawlers:
if(Request.Browser.Crawler)
if its a crawler (e.g. GoogleBot) manually bypass the age verification