Prevent spiders from following is matches URL

How do I prevent spriders crawling pages that start with mydomain.com/abc...

For example mydomain.com/abcSGGSHS or mydomain.com/abc6bNNha

I think I need to add some sort of regul开发者_运维问答ar expression to the web root's .htaccess, right?

With mod_rewrite enabled, you can do the following

RewriteEngine on 
RewriteBase / 
RewriteCond %{HTTP_USER_AGENT} ^YourBadSpiderName [OR]
RewriteCond %{HTTP_USER_AGENT} ^AotherBadSpider
RewriteCond %{REQUEST_URI} ^abc
RewriteRule ^$ http://mydomain.com/404.html [NC,L]

You'll have to update the spider names accordingly. If a bot changes his user agent, let's say to 'Mozilla/Firefox', you're out of luck..