Search code examples
securitysearch-engineweb-crawler

Can a secret URL be indexed by search engines?


My boss recently came to me with a security concern.

My company does research and then sends a report to the client (this report is unique for each client and contains identifying information about their business).

We email the report to the client, but we also upload it to our server and provide the client a url containing a hash that he can use to access the report online at any time. (in other words, anyone with the link can access the report) The link looks something like:

The code behind the page does the following:

  • explodes the hash and uses it to look up the report info in the database
  • verifies that this hash is indeed valid according to report id and email
  • retrieves the report pdf from the server and displays it.

If the hash in the url does not exist or is not valid, the user get an "invalid" message.

His question is:

Clicking on the link opens the report, does this mean that Google can index these pages and they may appear in Google's search results for the whole world to see?

My gut feeling is that no, if a crawler would get to report_page.php, all it would see if the invalid message, unless it somehow had a valid hash, but I don't know this for sure, and after searching around, I didn't come up with anything to prove me right (or wrong).

Any thoughts? Thanks!


Solution

  • If the URL is posted on a publically accessible page, then the Google crawlers will find it and the page will be indexed. From what you say, this sounds unlikely to be the case so you should be safe. If your client happens to make the URL public though, then Google will crawl and index it. It's similar to a password: keep it private and you're safe.