Search code examples
web-crawlergooglebot

How does google web crawler find my website if I didn't post url anywhere?


wondering how does the google web crawler find my website. Assuming I haven't posted the url in any other place indexed by google.

Can someone explain?


Solution

  • If you're site is hosted (like many other internet sites) inside a public hosting site company/hosting server farms, then these companies usually publish a main crawling index of all url's site hosted addresses that hints google bot on which sites it should be crawled/recrawled and which addresses were just created on their hosting server.

    Once informed Google use software known as “web crawlers” to discover publicly available webpages. The most well-known crawler is called “Googlebot.” Crawlers look at webpages and follow links on those pages, much like you would if you were browsing content on the web. They go from link to link and bring data about those webpages back to Google’s servers.