Search code examples
searchsearch-enginebotssearch-engine-bots

How to stop all search engines, bots to crawl some urls

I want to count ads click on a widget.

I've used on robot.txt file:

User-agent: *
Allow: /
Disallow: */ads_count/*

I've also add nofollow for all links in that widget.

But many bots still follow urls in that widget. I've get client ip to count urls and i got many IP form bots.

Solution

  • Did u try removing the (*) before */ads_count? As google documentation for SEO says, if you want to block all the bots, it's like u did:

    User-agent: * // (to whom? (*) means all bots!
Disallow: /ads_count

    Note that directives are case-sensitive. For instance, Disallow: /junk_file.asp would block http://www.example.com/junk_file.asp, but would allow http://www.example.com/Junk_file.asp. Googlebot will ignore white-space (in particular empty lines)and unknown directives in the robots.txt.