Search code examples
indexingseorobots.txtsearch-engine-bots

Block certain Addresses from being Crawled Robots.txt


I've been asked (in order to improve SEO results) to remove a certain web address from a Robots.txt.

I'm struggling (SEO is not my forte) to figure out how to block a certain domain, even though both addresses point to the same website.

For Example:

http://foo.example.com -- Web Address I need to stop from being Indexed
http://www.examplefoo.com -- Web Address I want to be indexed

From What I can tell, using a disallow / will only block from both (as they are the same site)

Any help?! Sorry if this is a silly question..


Solution

  • Step 1: If your first domain in up, you should redirect it to your second domain.

    Sample (width htaccess):

    RewriteCond %{HTTP_HOST} ^www.xxx.ir$ [NC,OR]
    
    RewriteCond %{HTTP_HOST} ^xxx.ir$ [NC,OR]
    
    RewriteCond %{HTTP_HOST} ^yyy.com$ [NC]
    
    RewriteRule ^(.*)$ http://www.yyy.com/$1 [R=301,L]
    

    Step 2: You should remove all urls (from first domain) by remove urls in webmaster tools (directory all)

    good luck