How to identify the client is a search robot?
I have built my entire site using AJAX (indeed it's GWT). I have also implemented AJAX crawling proposed by Google. However, after the implementation, I found that neither Yahoo , Bing, nor Baidu implemented that scheme!
I'm wondering if there is a way to identify the web client is a search robot. If they are, they will be shown the HTML snapshot I created.
开发者_如何学运维It will be best if I can identify them in APACHE level, then I can just do a mod_rewrite. But it's still ok if I can do that in PHP or GWT.
It's quite tricky as there are so many different search engines. I guess this can't be done in a one liner. If PHP is ok, I suggest using php_browscap.ini (available for different languages). This can be used with PHPs get_browser()-function, for example.
Then you can check against all different browser stuff and of course Bots.
In my own code snippets I use the following function (based on the mentioned browscap) to get the appropriate infos easily in my code:
function _browser($a_browser = false, $a_version = false, $name = false)
{
$browser_list = 'msie firefox konqueror safari netscape navigator opera mosaic lynx amaya omniweb chrome avant camino flock seamonkey aol mozilla gecko';
$user_browser = strtolower($_SERVER['HTTP_USER_AGENT']);
$this_version = $this_browser = '';
$browser_limit = strlen($user_browser);
foreach ($this->_w($browser_list) as $row)
{
$row = ($a_browser !== false) ? $a_browser : $row;
$n = stristr($user_browser, $row);
if (!$n || !empty($this_browser)) continue;
$this_browser = $row;
$j = strpos($user_browser, $row) + strlen($row) + 1;
for (; $j <= $browser_limit; $j++)
{
$s = trim(substr($user_browser, $j, 1));
$this_version .= $s;
if ($s === '') break;
}
}
if ($a_browser !== false)
{
$ret = false;
if (strtolower($a_browser) == $this_browser)
{
$ret = true;
if ($a_version !== false && !empty($this_version))
{
$a_sign = explode(' ', $a_version);
if (version_compare($this_version, $a_sign[1], $a_sign[0]) === false)
{
$ret = false;
}
}
}
return $ret;
}
//
$this_platform = '';
if (strpos($user_browser, 'linux'))
{
$this_platform = 'linux';
}
elseif (strpos($user_browser, 'macintosh') || strpos($user_browser, 'mac platform x'))
{
$this_platform = 'mac';
}
else if (strpos($user_browser, 'windows') || strpos($user_browser, 'win32'))
{
$this_platform = 'windows';
}
if ($name !== false)
{
return $this_browser . ' ' . $this_version;
}
return array(
"browser" => $this_browser,
"version" => $this_version,
"platform" => $this_platform,
"useragent" => $user_browser
);
}
function _w($a = '')
{
if (empty($a)) return array();
return explode(' ', $a);
}
Examples:
/*
// Examples
echo '<pre>';
print_r(_browser()); // return array of browser data
var_dump(_browser('firefox')); // return true if using firefox
var_dump(_browser('msie', '>= 7.0')); // return true if using IE 7.0 or above else false
var_dump(_browser('firefox', '< 3.0.5')); // return true if using below firefox 3.0.5 (can check minor version)
var_dump(_browser(false, false, true)); // return string of name of browser and version
// To check if Gecko browser is used
var_dump(_browser('gecko'));
// version_compared function is used so you can use the same operator syntax
var_dump(_browser('firefox', 'le 1.5'));
echo '</pre>';
To do it with Apache you can use a rewriteRule
together with a RewriteCond
on %{HTTP_USER_AGENT}
.
The RewriteCond accepts RegExp, so you have to use a pattern there, which matches all those bots(you should find informations how to build the pattern on the page linked by acme).
But be careful: Search-engines may penalize pages that deliver different contents to the bots and to the others clients.
If you use PHP, you can check $_SERVER['HTTP_USER_AGENT']
against the string of search engine User Agent string -- some details are found here: http://en.wikipedia.org/wiki/Web_crawler
You can also use a Firefox plugin to see how the search engines will see your page: http://chrispederick.com/work/user-agent-switcher/
精彩评论