should i then do 开发者_运维知识库User-agent: * Disallow: / is it as simple as that? or will that not crawl the files in the root either?
I\'m creating a forum for a website, and plan on implementing a \"Report this content\" function. In all honesty, I\'m not sure how useful (lit. necessary) the feature will be, since a user account (
I have a site which has a whole host of legacy links, which now need to be ma开发者_C百科pped to new links.
I am using htaccess in my site, such that all the request to my site will be redirected to index page in开发者_运维百科 my root directory. No other file in my site can be accessed because my htaccess
I have a large directory of individual names along with generic publicaly available and category specific information that I want indexed as much as possible in search engines. Listing these names on
i\'m using a online sitemap generator tool which generates links even for which are blocked in robots.txt. Is these block开发者_StackOverflow社区ed links affect site ranking ? . Is there anyway to ove
If i want to only allow crawlers to access index.php, will this 开发者_JAVA技巧work? User-agent: *
I\'ve got a Plone site that I administer and I\'d like to add some pages to the Disallow of a robots.txt.
Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow.
加载中,请稍侯......