How to know whether an incoming request is from a search engine robot?

Does anyone know how to detect whether an incoming开发者_如何学Python request is from a search engine robot? Do HTML headers contain any specific info for that?

You can usually use a combination of IP and the user agent in the HTTP headers to figure it out. There is an article all about it here: http://www.jafsoft.com/searchengines/spider_hunting.html

HTTP request has a field named 'user agent'. This field specifies if the request came from Mozilla or internet explorer or some search engine bot. Java, php each has its own command to read this user agent field.

You can use jQuery to check the user agent from the request.

You can get to this information through $.browser

Keep in mind, though, that this is easily spoofed.