Robots.txt priority question
If I have these lines in robots.txt:
Disallow /folder/
Allow /folder/filename.php
Will the filename.php be allowed then?
Which order does google pri开发者_运维问答oritize the lines?
What will happen here for example?:
Allow /
Disallow /
I am mainly referring to Google here as a search engine.
Thanks
For your first example, yes, the file /folder/filename.php
will be allowed, as Google will prioritize the most specific rule. See here or check Wikipedia's article, for example, for more details.
Furthermore, Google has a detailed documentation on how it uses robots.txt
over here.
Edit : For the second example, the Allow
will be prioritized as per the Wikipedia quote Byron Whitlock posted in his answer.
It depends on the search engine:
While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that Allow patterns with equal or more characters in the directive path win over a matching Disallow pattern. Bing uses the Allow or Disallow directive which is the most specific.
In order to be compatible to all robots, if one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow.
http://en.wikipedia.org/wiki/Robots_exclusion_standard
精彩评论