Ignoring lookups for google and search bots

[DJB31st]

I am using your code on one of the sites that i manage.

It occurred to me that there is no point in doing the ip lookup when a bot is crawling your site for a number of reasons.

The crawlers are usually always shown as US IP addresses

The crawlers don't set cookies so every page request will be a new IP request.

I think it would save a lot of lookups and also increase the perceived speed of the website to the search bots if the lookup was ignored or set to US whenever a bot request was made.

Am working on a mod to the default class now

[marchost]

Hi sorry for my late answer, I was away for some time.

yes indeed you could save queries if you manage to identify bots...

marc

[DJB31st]

Added the function below into the class

function detect_bot() {
$bot_list = array("Teoma", "alexa", "froogle", "Gigabot", "inktomi",
"looksmart", "URL_Spider_SQL", "Firefly", "NationalDirectory",
"Ask Jeeves", "TECNOSEEK", "InfoSeek", "WebFindBot", "girafabot",
"crawler", "www.galaxy.com", "Googlebot", "Scooter", "Slurp",
"msnbot", "appie", "FAST", "WebBug", "Spade", "ZyBorg", "rabaz",
"Baiduspider", "Feedfetcher-Google", "TechnoratiSnoop", "Rankivabot",
"Mediapartners-Google", "Sogou web spider", "WebAlta Crawler", "Yandex");

foreach($bot_list as $bot) {
if(ereg($bot, $_SERVER['HTTP_USER_AGENT'])) {
return $bot;
}

And put an if statement in the getGeoLocation function.

Line 249

if(detect_bot()=="") {

Line 297

}

Appears to have done the trick :-)

[jts]

I agree, and do something similar on my own site. I think it's in everyone's benefit to pass on the IPinfoDB.com API lookup if it's a bot/crawler on your site.

-jts