1 Topic by Programming-Designs (edited by Programming-Designs 2005-12-01 02:07)

  • Registered: 2005-11-11
  • Posts: 122

Topic: BotDetect Mod Release v1.5

##
##
##       Mod title:  BotDetect
##
##       Mod version:  1.5
##       Works on PunBB:  1.2.10, 1.2.*
##       Release date:  2005-11-29
##       Author:  Daniel Moxon (services@programming-designs.com)
##       Website: http://programming-designs.com
##
##       Description:  This Mod allows you to detect when a bot is online.
##
##       Affected files:  index.php
##
##
##       Affects DB:  Yes - tables added: bots, botsconfig
##
##
##
##       Notes: This is a DRAMATIC update from my last version -- it now
##       supports integration with the SQL database and allows custom
##       configuration from the Administrative Plug-in options. Quick and
##       easy to install.
##
##
##
##       DISCLAIMER:  Please note that "mods" are not officially supported by
##                    PunBB. Installation of this modification is done at your
##                    own risk. Backup your forum database and any and all
##                    applicable files before proceeding.
##
##
##

If you're interested in downloading it, click *here*.

Documentation on how to upgrade/install is provided at the given link.

For detecting bots on all pages of the forums see my response: http://punbb.org/forums/viewtopic.php?pid=58076#p58076


I found another potential problem. When case sensitive is not selected, that means if you type the bot string in upper case then it potentially could have problems as it would lowercase the useragent but NOT the user agent string you defined. I now updated it in the same zip file download. Go to the link I provided and reupload the botdetect.php file and that should do the trick -- this could make a LOT of the bots show up now.

Hope you guys like it smile

2 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

Couple things:

1. there is a case sensitivity issue throughout the plugin and install_mod.php where you seem to mix and match 

botsConfig

and 

botsconfig

this prevents the install of the db sql, and also prevents the plugin from running.

changing everything to "botsConfig" allows both installation and plugin to run.

2. The bot name still doesn't show up in the who is online list...

bummer.

3 Reply by Programming-Designs (edited by Programming-Designs 2005-11-30 08:16)

  • Registered: 2005-11-11
  • Posts: 122

Re: BotDetect Mod Release v1.5

- Fixed 1st issue by simply renaming botsConfig to botsconfig.
- Is very weird that it isn't showing up -- did you use the googlebot spoofer and try viewing your index.php page? I've got it running on my forums fine.

If the default bot definitions weren't added, you should add them in -- for Googlebot you could put Googlebot for the alias and googlebot for the User Agent String. Also make sure bot detection is enabled. I don't know what else could be causing the problems.

4 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

Yes, i did. No fopen() error this time either... just nothing showing up in the who list.

One thing:
I elected to change everything to "botsConfig" as opposed to botsconfig. Everything seems to be running fine. Shall I redo it the other way around and see if that works?

  • Registered: 2005-11-11
  • Posts: 122

Re: BotDetect Mod Release v1.5

I would give that a go, in my database the table is lower-cased.

6 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

That did it.

changed everything to the lower case: botsconfig

They show up when i go look through the link provided to test...

I do arin whois on the other 18 'guests' and determine that 14 of them are bots. They don't appear as multiple googlebots and what not. Will this only work when they arrive?

7 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

Guest's have increased now from 18 to 26, looks like 6 new bots, and no notice of them in the who is online list. Wonder if new aliases and id strings are required. I'll add some and see what happenes.

8 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

interesting... nothing changing. Do the bots appear and disappear on your who list on their own?

  • Registered: 2005-11-11
  • Posts: 122

Re: BotDetect Mod Release v1.5

dss -

This mod was made specifically to test the index page. It just occurred to me last night that I could have written the mod to allow bots to be detected on all pages.

If you wish to do so, it is quite simple really. Open up your 'index.php' file and do the following:

FIND AND REMOVE THIS:

require('botdetect.php');
    $botStatus = isBotOnline();

Now open up include/common.php and go to the end of the file and place the text back in:

require('botdetect.php');
    $botStatus = isBotOnline();

Upload both and now bots can be detected on any page.

I will write a re-write for my installation guide later with these new steps included.

10 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

It's working!

Notice the credit added in my footer for you.

11 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

One thing is that not all bots in the example show up. I have been looking for a reliable list of id strings to create an up to date list of bots and how they identify themselves.

Any idea where one might get a current list of this?

  • Registered: 2005-11-11
  • Posts: 122

Re: BotDetect Mod Release v1.5

I will have a look in a couple hours -- I have an idea where it may be but not sure, will check into it. May write an install_mod to delete/append a new list of bots when I have found a reliable source.

But, until then, I have something that may/may not interest you:

Color Code Your Bot
To color code your bot simply delete the bot definition (remember to hold onto the user agent string) and create a new one with this alias:

<dd style="color:#ff0000">Bot Alias Here</dd>

You can also use these for bold/italic/underlined text:

<b> </b> <i> </i> <u> </u>

This can differentiate the bots from the actual members or from other bots. Later on I made include features that automatically set all (or selected) bot's text formatting to what you choose.

13 Reply by Programming-Designs (edited by Programming-Designs 2005-12-01 02:07)

  • Registered: 2005-11-11
  • Posts: 122

Re: BotDetect Mod Release v1.5

dss, see this page on wikipedia: http://en.wikipedia.org/wiki/User_agent#Bots

Also scroll down and see external links.

dss -- I found another potential problem. When case sensitive is not selected, that means if you type the bot string in upper case then it potentially could have problems as it would lowercase the useragent but NOT the user agent string you defined. I now updated it in the same zip file download. Go to the link I provided and reupload the botdetect.php file and that should do the trick -- this could make a LOT of the bots show up now.

14 Reply by dss

  • dss
  • Senior Member
  • Offline
  • Registered: 2005-01-09
  • Posts: 119

Re: BotDetect Mod Release v1.5

Sorted: working great now!

  • Registered: 2005-12-12
  • Posts: 4

Re: BotDetect Mod Release v1.5

Wow that is pretty cool. Yeah I saw this on phpbb forums but never saw a feature like that in punbb. nice job

16 Reply by MrMister (edited by pogenwurst 2006-09-17 18:50)

  • Registered: 2006-03-12
  • Posts: 29

Re: BotDetect Mod Release v1.5

If anyone is interested here's a list of useragents and their strings.
I haven't verified all of them to check if they are correct, but it's a start smile

INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ichiro', 'ichiro/2.0 (ichiro@nttr.co.jp)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Alta Vista', 'Scooter/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Hogsearch', 'oegp v. 1.3.0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Kraehe [Metasuche]', '-DIE-KRAEHE- META-SEARCH-ENGINE/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo', 'Yahoo! Slurp');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google', 'Googlebot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Alexa', 'ia_archiver');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSN', 'msnbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] W3C [Validator]', 'W3C_Validator');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Ask Jeeves', 'Ask Jeeves');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] TurnitinBot', 'TurnitinBot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Seekport', 'Seekbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IBM Research', 'ibm.com/cs/crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo MMCrawler', 'Yahoo-MMCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Ocelli', 'Ocelli/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Gigabot', 'Gigabot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Majestic-12', 'MJ12bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Adsense', 'Mediapartners-Google/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Walhello', 'appie 1.1 (www.walhello.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Voyager', 'voyager/1.0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WiseNut', 'http://www.WISEnutbot.com');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Thumbshots [Capture]', 'thumbshots-de-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GeoBot', 'GeoBot/version');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] suchpad', 'http://www.suchpad.de/bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Telekom', 'crawleradmin.t-info@telekom.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Insuranco', 'InsurancoBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Xaldon [Spider]', 'Xaldon WebSpider');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ICCenter', 'iCCrawler (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nutch', 'Nutch; http://lucene.apache.org/nutch/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Cosmix', 'cfetch/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Heise IT-Markt [Crawler]', 'heise-IT-Markt-Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Esperanza', 'EsperanzaBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] EliteSys', 'EliteSys SuperBot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] NG-Search', 'http://www.ng-search.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] genie', 'genieBot (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MP3-Bot', 'MP3-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Sensis [Crawler]', 'Sensis Web Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Seoma [Crawler]', 'Seoma [SEO Crawler]');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] g2 [Crawler]', 'g2Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] OmniExplorer', 'OmniExplorer_Bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Baidu [Spider]', 'Baiduspider+(');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] W3C [Linkcheck]', 'W3C-checklink/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Accoona', 'Accoona-AI-Agent/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ichiro [Crawler]', 'ichiro/2');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GBSpider [Spider]', 'GBSpider v');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] psbot [Picsearch]', 'psbot/0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] PlantyNet', 'PlantyNet_WebRobot_V');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Twiceler', 'Twiceler www.cuill.com/robots.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IPG', 'internet-provider-guenstig.de-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Susie [Sync]', '!Susie (http://www.sync2it.com/susie)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WissenOnline', 'WissenOnline-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] 24spider', '24spider-Robot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Snappy', 'Snappy/1.1 ( http://www.urltrends.com/ )');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Zerx', 'zerxbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] LinkWalker', 'LinkWalker');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Exabot', 'Exabot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Jyxobot', 'Jyxobot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Tbot', 'Tbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Findexa [Crawler]', 'Findexa Crawler (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ISC Systems', 'ISC Systems iRc Search');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IRLbot', 'http://irl.cs.tamu.edu/crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Mirago', 'HeinrichderMiragoRobot (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Sygol', 'SygolBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nutch/CVS', 'NutchCVS/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WWWeasel', 'WWWeasel Robot v');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Naver', 'nhnbot@naver.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Heritrix [Crawler]', 'heritrix/1.');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] FAST Enterprise [Crawler]', 'FAST Enterprise Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Desktop', 'Google Desktop');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MMSBot', 'http://www.mmsweb.at/bot.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GeigerzaehlerBot', 'http://www.geigerzaehler.org/bot.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Vagabondo [Crawler]', 'http://webagent.wise-guys.nl/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nimble [Crawler]', 'NimbleCrawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Bunnybot', 'powered by www.buncat.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] YahooSeeker', 'YahooSeeker/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Boitho [Crawler]', 'boitho.com-dc/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Scumbot', 'Scumbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Steeler [Crawler]', 'http://www.tkl.iis.u-tokyo.ac.jp/~crawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] FAST WebCrawler [Crawler]', 'FAST-WebCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Orbiter', 'http://www.dailyorbit.com/bot.htm');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ASPseek [Crawler]', 'ASPseek/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Crawler Search [Crawler]', '.Crawler-Search.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Francis', 'http://www.neomo.de/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Online link [Validator]', 'online link validator');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Asterias', 'asterias/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Asterias [Spider]', 'http://www.singingfish.com/help/spider.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] NetResearchServer', 'NetResearchServer/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Orange [Spider]', 'OrangeSpider');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSNbot Media', 'msnbot-media/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] McSeek', 'powered by www.McSeek.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Webmeasurement', 'webmeasurement-bot,');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] 123spider', '123spider-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Cometrics', 'cometrics-bot,');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSN NewsBlogs', 'msnbot-NewsBlogs/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Houxou [Crawler]', 'HouxouCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Feedfetcher', 'Feedfetcher-Google');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yacy', 'yacybot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo Slurp', 'Yahoo! DE Slurp');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] AdsBot [Google]', 'AdsBot-Google');

17 Reply by Mark

  • Mark
  • PunBB Donor
  • Offline
  • From: England, North West
  • Registered: 2004-09-21
  • Posts: 834

Re: BotDetect Mod Release v1.5

please use [ code ] [/code ]  for the previous code

Mark's Website

18 Reply by no64

  • no64
  • Senior Member
  • Offline
  • Registered: 2007-06-18
  • Posts: 108

Re: BotDetect Mod Release v1.5

Is this working with version 1.2.15 ?

  • Registered: 2006-12-11
  • Posts: 34

Re: BotDetect Mod Release v1.5

How can i fix so it working on punBB 1.2.21 ???

http://www.pokemonforumet.se/forum/index.php