robots.txt and seo ranking
is there any point in having a robots.txt with just one line of content like below
User-agent: *
开发者_高级运维compared to not having it at all ?
No; there is no point in doing that.
No, google (and let's not beat around the bush, Google is the search engine you should care about) by default will crawl everything unless you tell it not to.
If you want to help Google, use google webmaster tools and submit a sitemap in the correct format
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184
精彩评论