Search code examples
asp.netsearch-engineweb-crawler

Robots.txt with one site but two domains

I have a website which has two domains added. Both domains point to the root of the website. Is it possible to alter the robots.txt so that one of the domains doesn't get crawled, while the other still does?

Solution

  • You ought to perform a 301 redirect to your preferred domain, which tells robots what URL you want crawled.

    For example

    www.domain1.com

    Should 301 redirect to

    www.domain2.com

    Similarly

    www.domain1.com/Contact-Us/

    Should 301 redirect to

    www.domain2.com/Contact-Us/