开发者

robots.txt and Mod Rewrite in .htaccess

In the robots.txt file, I am about to disallow some sections of my site.

For instance, I don't want my "terms and conditions" to be indexed by search engines.

 User-agent: *
 Disallow: /terms

The real path to the file is actua开发者_如何学Pythonlly

 /data/terms_and_conditions.html

But I have used .htaccess to rewrite the URL.

Now to my Q, should I specify the rewritten url in the robots.txt or the actual url?

Follow-up question: Do I need to have an "allow" line too, or will the search engines assume all other is allowed which isn't in the robots.txt file?

Thanks


Search engines will assume that all other is allowed which isn't in the robots.txt. In your case it will disallow path /term.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜