Search code examples
web-crawlerrobots.txtmeta-tagsnoindex

Can I store a page with the NOINDEX metatag?


I'm doing a webcreawler and now I'm trying to follow the robots.txt and the metatags NOINDEX and NOFOLLOW.

NOFOLLOW says me to not follow the links in the page, but and the NOINDEX?
I understand that it's to not index the page on the searches, but I won't do any search, only store the page in the database.

I will disrespect the NOINDEX tag if I store in the database or this only applies to searches?


Solution

  • No particular reason not to store it. As long as you're not indexing it and returning it in a search result. You're not disrespecting the meaning of NOINDEX unless you return it as a search result.