Prevent spiders from following is matches URL
How do I prevent spriders crawling pages that start with mydomain.com/abc...
For example mydomain.com/abcSGGSHS or mydomain.com/abc6bNNha
I think I need to add some sort of regul开发者_运维问答ar expression to the web root's .htaccess, right?
With mod_rewrite enabled, you can do the following
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^YourBadSpiderName [OR]
RewriteCond %{HTTP_USER_AGENT} ^AotherBadSpider
RewriteCond %{REQUEST_URI} ^abc
RewriteRule ^$ http://mydomain.com/404.html [NC,L]
You'll have to update the spider names accordingly. If a bot changes his user agent, let's say to 'Mozilla/Firefox', you're out of luck..
精彩评论