Search code examples
search-enginerobots.txtgooglebot

How do I use a wildcard to block bots from indexing this?

I have the following URL format:

example.com/page/m6aoeh/embed

I want to block any bots from indexing any page with the url /page/*/embed

I'm assuming there's some way to do wildcard URL block, but just not sure how to do it in this particular case.

Solution

  • If you simply want to indicate to bots that everything under the /page/ folder should not be indexed by bots, use the robots.txt exclusion standard. Create a /robots.txt that looks like this:

    User-agent: *
Disallow: /page/

    If you want bots to index some pages under the /page folder but not others, you'll need to add a <meta> tag in the <head> to each page you want to exclude.

    <meta name="robots" content="noindex, nofollow" />