Search code examples
php.htaccessindexingsearch-enginekohana

.htaccess close for search engines whole site, except some urls (kohana)

About:
Site in kohana, version 3.0.8.

Problem:
I want to close for search engines whole site, except
baseinvest.kz/project and inner pages of it(baseinvest.kz/project/view/110)

.htaccess file

RewriteEngine On
RewriteBase /
<Files .*>
Order Deny,Allow
Deny From All
</Files>
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]
RewriteRule ^(?:application|modules|system)\b.* index.php/$0 [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteCond %{REQUEST_URI} !=/robots.txt
RewriteRule .* index.php [L]
Solution

  • I've found solution by adding file robots.txt

    User-agent: *
Disallow: /
Allow: /project

    Analyzed by this site http://tools.seobook.com/robots-txt/analyzer/