Search code examples
.htaccesssearch-enginerobots.txt

robots.txt htaccess block google

In my .htaccess file I have:

<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
</Files>

This denies any text file from being read, but the Google search engine gives me the following error:

robots.txt Status

http://mysite/robots.txt

18 minutes ago  302 (Moved temporarily)

How can I modify .htaccess to permit Google to read robots.txt while prohibiting everyone else from accessing text files?

Solution

  • Use this:

    <Files ~ "\.(tpl|txt)$">
  Order deny,allow
  Deny from all
  SetEnvIfNoCase User-Agent "Googlebot" goodbot
  Allow from env=goodbot
</Files>