Prevent search engines indexing pages with 3 or more forwarslashes in their paths
How can I prevent search engines indexing pages with 3 or more forw开发者_运维百科arslashes in their paths? for example:
www.example.com/about.html ->ok
www.example.com/1/2/3/4/5/test.html -> no index!!!
How should I write the robots.txt? Thanks!
Try with
User-Agent: *
Disallow: /*/*/*/*
Although I'm not sure all bots support the *
.
精彩评论