Search code examples
htmlwebsearch-engineduplication

How to solve duplicate content for search engines?

I have 2 websites: Site1.com and Site2.com.

Site1.com has 2 pages: a.htm and b.htm. Site2.com has 3 pages: a.htm, b.htm and c.htm.

Pages a.htm and b.htm are duplication at the 2 sites.

I want search engines to index a.htm and b.htm from Site1 and c.htm from Site2. How and where do I specify this preference? I prefer not to have to specify my preference at the page level (meaning having to edit every page) if possible.

Thanks.

Solution

  • Create a robots.txt file on the root of each server, specifying which pages should be indexed.

    For example, on Site 2:

    User-agent: *
Disallow: /a.htm
Disallow: /b.htm