Search code examples
search-enginerobots.txt

What does robots.txt file do in PHP project?

What does robots.txt file do in PHP project?

Solution

  • Not really PHP, see:

    Introduction to "robots.txt"

    "Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.