What is called first - robots.txt or mod_rewrite in htaccess
I need some help. I'm not sure about the order on request for mod开发者_运维问答_rewrite and robots.txt.
Some urls belong to a rewrite rule:
/index.php?id=123 to /home
Other urls don't have a rewrite:
/index.php?id=444
I made this entry to my robots.txt:
User-agent: *
Disallow: /index.php?id
Will the site with /home be indexed by search engines?
The robots.txt
file is interpreted by the client (spider), and they don't know what rewrites you have in your system. Thus, spiders would not fetch URLs from your site if they look like the pattern in robots.txt
but would if they found the same content through /home
.
精彩评论