Search code examples
seosearch-enginerobots.txtgoogle-crawlers

Disallow specific folders in robots.txt with wildcards

Can i hide specific folders from crawlers with wildcards like:

 User-agent: *
 Disallow: /system/
 Disallow: /v*

I want to hide all folders starts with "v" character. It will work this way?

Solution

  • You don't need wildcards at all for this. Your example will work, but it would work just as well without the wildcard. Trailing wildcards do not do anything useful.

    For example, this:

    Disallow: /x

    means: "Block any path that starts with '/x', followed by zero or more characters."

    And this:

    Disallow: /x*

    means: "Block any path that starts with '/x', followed by zero or more characters, followed by zero or more characters."

    This is redundant, and it blocks all of the same things the first one blocks. The only practical difference is that the second version will fail to work on crawlers that don't support wildcards.