How to ignore web crawlers?

I have a page that count how many times is visited by a user (registered, guest, every kind of users...).

So I update a field on the database every time the page is viewed; yes, also if the page is refreshed quickly, but I don't mind about this.

Of course, when some bots/crawler scans my website they will increment this value, and I'll get rid about this. So, is there a list of IP addresses to ignore? Or some mechanism that can he开发者_如何学编程lp me to do it?

Another way to do it is with ajax. Most crawlers don't parse javascript.

IP addresses can change so it's not be the best way to detect whether or not a visitor is a bot. Instead, I suggest looking at the user-agent string in the HTTP request parameters.

Here's a list of user-agent strings: http://www.user-agents.org/ . Look specifically under the type R for "robots, crawler, spider".

Most people don't have a static IP address. Have you setup a robots.txt to deny access to crawlers/bots? You could periodically query your log files to try and identify those which don't respect robots.txt, though the user agent is easily spoofed/changed.