Allow and disallow in robots.txt file
I want to disallow all files and folders on my site from SE bots, except a special folder and files in it. Can I use these lines at robots.txt file?
User-agent: *
Disallow: /
Allow: /thatfolder
Is it r开发者_StackOverflowight?
User-agent:*
Allow: /thatfolder
Disallow: /
This should do. (just changed the order)
精彩评论