Search code examples
securityiisdirectoryrights

Would search engines pick up files in IIS folder, that are not linked to any URL, and in Non-Browseable folder

We have an application that is hosted on IIS, and we have specific files in a folder that we do not want any search engine to register.

I have read this:

The robots.txt standard is meant for this. Example

User-agent: *
Disallow: /protected-directory/

I understand that this does not necessarily mean that the files won't be indexed.

My question: would the search engines pick up files that do not have any URL leading to those files? This directory is not browse-able, and there is no link to it. Am I safe in drop the file there, or would the search engine crawlers pick it up?

Solution

  • If you do not have this link posted somewhere on the Web, and your directory is non-browseable, then crawler will never find this file and will never index it