What does robots.txt file do in PHP project?

What does开发者_开发知识库 robots.txt file do in PHP project?

Not really PHP, see:

Introduction to "robots.txt"

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

robots.txt is related to search engines robots and not specific to PHP projects. it contains instructions for robots that crawl you website such as which urls you dont want to be crawled by the search engines and so on.

see http://www.webconfs.com/what-is-robots-txt-article-12.php

It is used to instruct webcrawlers like google and yahoo to crawl only parts of the page or not at all. See http://www.robotstxt.org for more information.

robots.txt lives in the / of your website.

When a search engine bot finds your website it first requests the robots.txt to find out which URIs it CAN and CANNOT index.

This is to prevent (it's a CONVENTION, so robots CAN ignore it) certain parts of your website from popping up in Google / Yahoo (for example your shoutbox...)

More info http://en.wikipedia.org/wiki/Robots_exclusion_standard