I'm doing a webcreawler and now I'm trying to follow the robots.txt and the metatags NOINDEX
and NOFOLLOW
.
NOFOLLOW
says me to not follow the links in the page, but and the NOINDEX
?
I understand that it's to not index the page on the searches, but I won't do any search, only store the page in the database.
I will disrespect the NOINDEX
tag if I store in the database or this only applies to searches?
No particular reason not to store it. As long as you're not indexing it and returning it in a search result. You're not disrespecting the meaning of NOINDEX
unless you return it as a search result.