BotDetect Mod Release v1.5

Programming-Designs · 2005-11-30 04:29:49

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

##
##
##       Mod title:  BotDetect
##
##       Mod version:  1.5
##       Works on PunBB:  1.2.10, 1.2.*
##       Release date:  2005-11-29
##       Author:  Daniel Moxon (services@programming-designs.com)
##       Website: http://programming-designs.com
##
##       Description:  This Mod allows you to detect when a bot is online.
##
##       Affected files:  index.php
##
##
##       Affects DB:  Yes - tables added: bots, botsconfig
##
##
##
##       Notes: This is a DRAMATIC update from my last version -- it now
##       supports integration with the SQL database and allows custom
##       configuration from the Administrative Plug-in options. Quick and
##       easy to install.
##
##
##
##       DISCLAIMER:  Please note that "mods" are not officially supported by
##                    PunBB. Installation of this modification is done at your
##                    own risk. Backup your forum database and any and all
##                    applicable files before proceeding.
##
##
##

If you're interested in downloading it, click *here*.

Documentation on how to upgrade/install is provided at the given link.

For detecting bots on all pages of the forums see my response: http://punbb.org/forums/viewtopic.php?pid=58076#p58076

I found another potential problem. When case sensitive is not selected, that means if you type the bot string in upper case then it potentially could have problems as it would lowercase the useragent but NOT the user agent string you defined. I now updated it in the same zip file download. Go to the link I provided and reupload the botdetect.php file and that should do the trick -- this could make a LOT of the bots show up now.

Hope you guys like it

Last edited by Programming-Designs (2005-12-01 02:07:45)

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

dss · 2005-11-30 07:59:18

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

Couple things:

1. there is a case sensitivity issue throughout the plugin and install_mod.php where you seem to mix and match

botsConfig

and

botsconfig

this prevents the install of the db sql, and also prevents the plugin from running.

changing everything to "botsConfig" allows both installation and plugin to run.

2. The bot name still doesn't show up in the who is online list...

bummer.

Programming-Designs · 2005-11-30 08:13:33

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

- Fixed 1st issue by simply renaming botsConfig to botsconfig.
- Is very weird that it isn't showing up -- did you use the googlebot spoofer and try viewing your index.php page? I've got it running on my forums fine.

If the default bot definitions weren't added, you should add them in -- for Googlebot you could put Googlebot for the alias and googlebot for the User Agent String. Also make sure bot detection is enabled. I don't know what else could be causing the problems.

Last edited by Programming-Designs (2005-11-30 08:16:59)

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

dss · 2005-11-30 08:15:42

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

Yes, i did. No fopen() error this time either... just nothing showing up in the who list.

One thing:
I elected to change everything to "botsConfig" as opposed to botsconfig. Everything seems to be running fine. Shall I redo it the other way around and see if that works?

Programming-Designs · 2005-11-30 08:20:00

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

I would give that a go, in my database the table is lower-cased.

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

dss · 2005-11-30 08:41:46

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

That did it.

changed everything to the lower case: botsconfig

They show up when i go look through the link provided to test...

I do arin whois on the other 18 'guests' and determine that 14 of them are bots. They don't appear as multiple googlebots and what not. Will this only work when they arrive?

dss · 2005-11-30 08:54:39

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

Guest's have increased now from 18 to 26, looks like 6 new bots, and no notice of them in the who is online list. Wonder if new aliases and id strings are required. I'll add some and see what happenes.

dss · 2005-11-30 09:22:00

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

interesting... nothing changing. Do the bots appear and disappear on your who list on their own?

Programming-Designs · 2005-11-30 14:52:36

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

dss -

This mod was made specifically to test the index page. It just occurred to me last night that I could have written the mod to allow bots to be detected on all pages.

If you wish to do so, it is quite simple really. Open up your 'index.php' file and do the following:

FIND AND REMOVE THIS:

require('botdetect.php');
$botStatus = isBotOnline();

Now open up include/common.php and go to the end of the file and place the text back in:

require('botdetect.php');
$botStatus = isBotOnline();

Upload both and now bots can be detected on any page.

I will write a re-write for my installation guide later with these new steps included.

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

dss · 2005-11-30 21:45:31

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

It's working!

Notice the credit added in my footer for you.

dss · 2005-11-30 22:07:52

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

One thing is that not all bots in the example show up. I have been looking for a reliable list of id strings to create an up to date list of bots and how they identify themselves.

Any idea where one might get a current list of this?

Programming-Designs · 2005-11-30 22:19:59

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

I will have a look in a couple hours -- I have an idea where it may be but not sure, will check into it. May write an install_mod to delete/append a new list of bots when I have found a reliable source.

But, until then, I have something that may/may not interest you:

Color Code Your Bot
To color code your bot simply delete the bot definition (remember to hold onto the user agent string) and create a new one with this alias:

<dd style="color:#ff0000">Bot Alias Here</dd>

You can also use these for bold/italic/underlined text:

<b> </b> <i> </i> <u> </u>

This can differentiate the bots from the actual members or from other bots. Later on I made include features that automatically set all (or selected) bot's text formatting to what you choose.

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

Programming-Designs · 2005-12-01 01:56:48

Programming-Designs
Senior Member
Offline

Registered: 2005-11-11
Posts: 122

Re:

dss, see this page on wikipedia: http://en.wikipedia.org/wiki/User_agent#Bots

Also scroll down and see external links.

dss -- I found another potential problem. When case sensitive is not selected, that means if you type the bot string in upper case then it potentially could have problems as it would lowercase the useragent but NOT the user agent string you defined. I now updated it in the same zip file download. Go to the link I provided and reupload the botdetect.php file and that should do the trick -- this could make a LOT of the bots show up now.

Last edited by Programming-Designs (2005-12-01 02:07:19)

Programming Designs - Tech, programming, and web design blog
PD Forums - Using the latest and greatest PunBB 1.3 =)

dss · 2005-12-01 17:34:25

dss
Senior Member
Offline

Registered: 2005-01-09
Posts: 119

Re:

Sorted: working great now!

jason117 · 2005-12-12 03:27:19

jason117
New member
Offline

Registered: 2005-12-12
Posts: 4

Re:

Wow that is pretty cool. Yeah I saw this on phpbb forums but never saw a feature like that in punbb. nice job

MrMister · 2006-09-17 15:47:52

MrMister
Member
Offline

Registered: 2006-03-12
Posts: 29

Re:

If anyone is interested here's a list of useragents and their strings.
I haven't verified all of them to check if they are correct, but it's a start

INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ichiro', 'ichiro/2.0 (ichiro@nttr.co.jp)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Alta Vista', 'Scooter/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Hogsearch', 'oegp v. 1.3.0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Kraehe [Metasuche]', '-DIE-KRAEHE- META-SEARCH-ENGINE/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo', 'Yahoo! Slurp');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google', 'Googlebot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Alexa', 'ia_archiver');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSN', 'msnbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] W3C [Validator]', 'W3C_Validator');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Ask Jeeves', 'Ask Jeeves');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] TurnitinBot', 'TurnitinBot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Seekport', 'Seekbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IBM Research', 'ibm.com/cs/crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo MMCrawler', 'Yahoo-MMCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Ocelli', 'Ocelli/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Gigabot', 'Gigabot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Majestic-12', 'MJ12bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Adsense', 'Mediapartners-Google/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Walhello', 'appie 1.1 (www.walhello.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Voyager', 'voyager/1.0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WiseNut', 'http://www.WISEnutbot.com');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Thumbshots [Capture]', 'thumbshots-de-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GeoBot', 'GeoBot/version');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] suchpad', 'http://www.suchpad.de/bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Telekom', 'crawleradmin.t-info@telekom.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Insuranco', 'InsurancoBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Xaldon [Spider]', 'Xaldon WebSpider');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ICCenter', 'iCCrawler (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nutch', 'Nutch; http://lucene.apache.org/nutch/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Cosmix', 'cfetch/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Heise IT-Markt [Crawler]', 'heise-IT-Markt-Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Esperanza', 'EsperanzaBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] EliteSys', 'EliteSys SuperBot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] NG-Search', 'http://www.ng-search.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] genie', 'genieBot (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MP3-Bot', 'MP3-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Sensis [Crawler]', 'Sensis Web Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Seoma [Crawler]', 'Seoma [SEO Crawler]');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] g2 [Crawler]', 'g2Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] OmniExplorer', 'OmniExplorer_Bot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Baidu [Spider]', 'Baiduspider+(');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] W3C [Linkcheck]', 'W3C-checklink/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Accoona', 'Accoona-AI-Agent/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ichiro [Crawler]', 'ichiro/2');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GBSpider [Spider]', 'GBSpider v');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] psbot [Picsearch]', 'psbot/0');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] PlantyNet', 'PlantyNet_WebRobot_V');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Twiceler', 'Twiceler www.cuill.com/robots.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IPG', 'internet-provider-guenstig.de-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Susie [Sync]', '!Susie (http://www.sync2it.com/susie)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WissenOnline', 'WissenOnline-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] 24spider', '24spider-Robot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Snappy', 'Snappy/1.1 ( http://www.urltrends.com/ )');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Zerx', 'zerxbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] LinkWalker', 'LinkWalker');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Exabot', 'Exabot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Jyxobot', 'Jyxobot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Tbot', 'Tbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Findexa [Crawler]', 'Findexa Crawler (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ISC Systems', 'ISC Systems iRc Search');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] IRLbot', 'http://irl.cs.tamu.edu/crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Mirago', 'HeinrichderMiragoRobot (');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Sygol', 'SygolBot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nutch/CVS', 'NutchCVS/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] WWWeasel', 'WWWeasel Robot v');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Naver', 'nhnbot@naver.com)');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Heritrix [Crawler]', 'heritrix/1.');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] FAST Enterprise [Crawler]', 'FAST Enterprise Crawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Desktop', 'Google Desktop');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MMSBot', 'http://www.mmsweb.at/bot.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] GeigerzaehlerBot', 'http://www.geigerzaehler.org/bot.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Vagabondo [Crawler]', 'http://webagent.wise-guys.nl/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Nimble [Crawler]', 'NimbleCrawler');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Bunnybot', 'powered by www.buncat.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] YahooSeeker', 'YahooSeeker/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Boitho [Crawler]', 'boitho.com-dc/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Scumbot', 'Scumbot/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Steeler [Crawler]', 'http://www.tkl.iis.u-tokyo.ac.jp/~crawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] FAST WebCrawler [Crawler]', 'FAST-WebCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Orbiter', 'http://www.dailyorbit.com/bot.htm');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] ASPseek [Crawler]', 'ASPseek/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Crawler Search [Crawler]', '.Crawler-Search.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Francis', 'http://www.neomo.de/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Online link [Validator]', 'online link validator');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Asterias', 'asterias/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Asterias [Spider]', 'http://www.singingfish.com/help/spider.html');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] NetResearchServer', 'NetResearchServer/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Orange [Spider]', 'OrangeSpider');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSNbot Media', 'msnbot-media/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] McSeek', 'powered by www.McSeek.de');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Webmeasurement', 'webmeasurement-bot,');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] 123spider', '123spider-Bot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Cometrics', 'cometrics-bot,');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] MSN NewsBlogs', 'msnbot-NewsBlogs/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Houxou [Crawler]', 'HouxouCrawler/');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Google Feedfetcher', 'Feedfetcher-Google');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yacy', 'yacybot');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] Yahoo Slurp', 'Yahoo! DE Slurp');
INSERT INTO punbb_bots (bot_alias, bot_string) VALUES ('[Bot] AdsBot [Google]', 'AdsBot-Google');

Last edited by pogenwurst (2006-09-17 18:50:28)

Mark · 2006-09-17 16:00:08

Mark
PunBB Donor
Offline

From: England, North West
Registered: 2004-09-21
Posts: 834

Re:

please use [ code ] [/code ] for the previous code

no64 · 2007-10-22 19:27:18

no64
Senior Member
Offline

Registered: 2007-06-18
Posts: 108

Re:

Is this working with version 1.2.15 ?

tokar86a · 2009-08-04 05:58:05

tokar86a
Member
Offline

Registered: 2006-12-11
Posts: 33

Re:

How can i fix so it working on punBB 1.2.21 ???

http://www.pokemonforumet.se/forum/index.php

BotDetect Mod Release v1.5

Posts [ 19 ]

1 Reply by Programming-Designs 2005-11-30 04:29:49

Re:

2 Reply by dss 2005-11-30 07:59:18

Re:

3 Reply by Programming-Designs 2005-11-30 08:13:33

Re:

4 Reply by dss 2005-11-30 08:15:42

Re:

5 Reply by Programming-Designs 2005-11-30 08:20:00

Re:

6 Reply by dss 2005-11-30 08:41:46

Re:

7 Reply by dss 2005-11-30 08:54:39

Re:

8 Reply by dss 2005-11-30 09:22:00

Re:

9 Reply by Programming-Designs 2005-11-30 14:52:36

Re:

10 Reply by dss 2005-11-30 21:45:31

Re:

11 Reply by dss 2005-11-30 22:07:52

Re:

12 Reply by Programming-Designs 2005-11-30 22:19:59

Re:

13 Reply by Programming-Designs 2005-12-01 01:56:48

Re:

14 Reply by dss 2005-12-01 17:34:25

Re:

15 Reply by jason117 2005-12-12 03:27:19

Re:

16 Reply by MrMister 2006-09-17 15:47:52

Re:

17 Reply by Mark 2006-09-17 16:00:08

Re:

18 Reply by no64 2007-10-22 19:27:18

Re:

19 Reply by tokar86a 2009-08-04 05:58:05

Re:

Posts [ 19 ]