for the past months, googleBot has been hitting a file that does not exist anymore on my site [index.php] as all the routing to the proper home pages in the proper languages is handled via apache rewrite rules in the htaccess.
Ans to, I commented out my .htaccess the DirectoryIndex index.php rule
RewriteEngine on
RewriteBase /
Options +FollowSymLinks -Indexes -ExecCGI
# DirectoryIndex index.php (not needed anymore, index.php doesnt exist)
# DirectoryIndex /en/home (should it be set to this now??)
Currently, Everything works sublime: the http://website.org root is redirected instantly to /en/home via 301 permanent rediret!
66.249.67.142 / == crawl-66-249-67-142.googlebot.com is hitting my site again and again trying to read index.php, which does not exist. What should I do??
A sneak peak into the endless error log file with such entries (poor googlebot i thought it might be more intelligent...)
[Fri Mar 04 20:48:30 2011] [error] [client 66.249.66.177] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
[Fri Mar 04 20:58:59 2011] [error] [client 66.249.66.177] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
[Fri Mar 04 21:00:18 2011] [error] [client 66.249.67.142] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
[Fri Mar 04 21:01:05 2011] [error] [client 66.249.66.177] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
[Fri Mar 04 21:12:28 2011] [error] [client 66.249.66.164] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
[Fri Mar 04 21:27:30 2011] [error] [client 66.249.68.115] File does not exist:
/var/www/vhosts/site.com/httpdocs/index.php
Someone linked to index.php, so Google is trying to follow it.
Do a rewrite from index.php to /en/home, and you'll be golden.
E: Also, DirectoryIndex can't be a Path, AFAIK. It just tells the server which file in a directory should be served if not specified otherwise.