开发者

Is noindex valid in robots.txt? [duplicate]

This question already has answers here: Noindex in a robots.txt 开发者_如何学JAVA (2 answers) Closed 1 year ago.

Is noindex an optional directive in a robots.txt file, or are user-agent, disallow, allow and crawl-delay the only options?

For example, is this valid for the contents of a robots.txt file?

user-agent: *  
disallow: /  
noindex: /


noindex is not a valid directive for a robots.txt file. It is a valid directive for a META robots tag, though.

The only standard directives for robots.txt are "User-agent" and "Disallow". Some browsers support an extended set of directives including "Crawl-delay", "Allow" and "Sitemap". http://rield.com/cheat-sheets/robots-exclusion-standard-protocol seems to have a thorough explanation of the standard and extended directives.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜