1 (edited by KeyDog 2010-09-26 16:21)

Topic: [Extension release] URL Checker For Posts And Signatures

Latest version

URL Checker v 0.5.2 (tested on 1.4.1 ! )



http://keydogbb.info/img/urlchecker.png

old
URL Checker v 0.5

2

Re: [Extension release] URL Checker For Posts And Signatures

Well, I could try to (well ok, I admit - I'm working on that right now, but I've got a little problem here which I probably leave on tomorrow) programme an extension, that would cross-check urls in posts and signatures smile

Eraversum - scifi browser-based online webgame

3

Re: [Extension release] URL Checker For Posts And Signatures

Ok, so problem fixed and extension seems to be working. It's cross-checking URLs you try to put in your signature or post.

http://dl.dropbox.com/u/6232434/PunBB/url_spam.PNG
It's checking only precise match with csv file (so the http://www.buildingquotes.co.uk/ isn't fetched as "spam" but the http://www.buildingquotes.co.uk/boiler-service/ is) (because of the youtube and so on wink)

Oh, and btw I changed the .csv file a little bit - I erased all white-space before quotes. I need that because I'm checking URL with those quotes (for exact match and not only part of it)

The only remaining problem is how to update the list? I think i'll have to check how works updating of official extensions  hmm  smile

Eraversum - scifi browser-based online webgame

4 (edited by KeyDog 2010-09-12 11:51)

Re: [Extension release] URL Checker For Posts And Signatures

Grez wrote:

Ok, so problem fixed and extension seems to be working. It's cross-checking URLs you try to put in your signature or post.

F-a-n-t-a-s-t-i-c   big_smile

Grez wrote:

Oh, and btw I changed the .csv file a little bit - I erased all white-space before quotes. I need that because I'm checking URL with those quotes (for exact match and not only part of it)


EDIT: Solved. The correct & newest one is now  HERE
(v89 - 12-09-2010)  Go here for latest version.

Grez wrote:

The only remaining problem is how to update the list? I think i'll have to check how works updating of official extensions  hmm  smile

Yes that will be a bit of a problem, as the official ext are on the informer server.... I guess if I can upload the file via ftp to the folder, then people can upgrade (manually) ...

I'm not sure yet, but I've noticed the urls change daily. The spammers are using fast flux etc and are extremly adaptive... I guess this will all be very "beta"...


EDIT: PS: I'm hoping that SFS will soon have a db that we could query. See discussion but would think it wouldn't be bad having an independent db backed by the community...

Re: [Extension release] URL Checker For Posts And Signatures

Improvement idea:
Like in stopforumspam extension;
insert a table that shows

  • what user (+IP) tried to submit an URL

  • which URL it was

Advantages:

  • possible to then ban users by cross checking with that new table

  • it's a good way to see if it's working

What do you think?

6 (edited by Grez 2010-08-07 09:24)

Re: [Extension release] URL Checker For Posts And Signatures

KeyDog wrote:

I'm not sure yet, but I've noticed the urls change daily. The spammers are using fast flux etc and are extremly adaptive... I guess this will all be very "beta"...

Damn it, how shifty they are!  hmm

KeyDog wrote:

Improvement idea:

Yup, I guess I could do that  smile

Eraversum - scifi browser-based online webgame

7

Re: [Extension release] URL Checker For Posts And Signatures

KeyDog wrote:

Like in stopforumspam extension; insert a table that shows

Done. Download link  wink

BTW I found nice bug with this topic big_smile You can't search it (I tried "checker" "url checker" but I didn't found anything - guess I'll have to look whether search uses caching), but more importantly, this topic didn't show up in topics list for this forum - check out images) big_smile

Page 1
Page 2
Topic's last post

Is splitting topics core part of PunBB or is it an extension? Just thinking whether it could have any connection smile

Eraversum - scifi browser-based online webgame

8 (edited by KeyDog 2010-08-16 17:58)

Re: [Extension release] URL Checker For Posts And Signatures

Grez wrote:

You can't search it

Strange, I see everything as I should...  AH HANG ON, I just split a topic off, and it doesn't show up either, you're right big_smile

EDIT: I just noticed that if the post that is being split is say 5 days old, the new split of topic will be down the list, when that post was made so 5 days ago....

Grez wrote:

Is splitting topics core part of PunBB or is it an extension?

Pun Move Posts 1.0.1, I believe....

Grez wrote:

Done. Download link  wink

Thanks a lot, will test... EDIT: Can confirm works as wanted ! Great stuff.
NOTE: I have incl. the latest spam.csv file in this release with over 1200 URIs

Re: [Extension release] URL Checker For Posts And Signatures

@Grez

Just found a new variation of url being entered

This inbetween url tags (the backslash being new)

\http://spamsitename.com/\

so

[ url ] http://thespam/ [/ url]

blocked


but if they use

[ url =\http://thespam/\] spam keyword [/ url]

they get past it ....



If I put the link in my .csv like that it causes  when I try to post a topic with that url
An error was encountered

Note: For detailed error information (necessary for troubleshooting), enable "DEBUG mode". To enable "DEBUG mode", open up the file config.php in a text editor, add a line that looks like "define('FORUM_DEBUG', 1);" (without the quotation marks), and re-upload the file. Once you've solved the problem, it is recommended that "DEBUG mode" be turned off again (just remove the line from the file and re-upload it).

Any ideas?

10

Re: [Extension release] URL Checker For Posts And Signatures

But with that the url is not working then, is it?

Actually purpose of this extension isn't imo to "block spambots" but to show them to admin, so he can ban + delete them. And I think the bot will firstly try "working" way, before trying backslashs and so  neutral

Eraversum - scifi browser-based online webgame

11

Re: [Extension release] URL Checker For Posts And Signatures

Intention I have:
Block the url being posted AND show to admin (as it doesn't ban the user)

Some bots immediately tried the backslashes.

So what would have to be changed to get the result:
1. Warning as is now
+ if you publish this URL your account will be blocked.
2. If user tries to publish even then:
You  tried to publish a URL on the spam list, you're account is temporarily blocked. A moderator will contact you shortly, ( for example)

Edit: Example - I've had the same spammer come and post 3-8 posts, all with the same 8 links. If that code in URL Checker blocked the user as soon as he confirmed he's wants to send those links, dozens of users would not get an email with his spam links. Very important in my eyes.

12

Re: [Extension release] URL Checker For Posts And Signatures

Can you add this URL to the list?  I've been getting spammed by it for a week now:

http://www.overstocksilver.com/sterling-silver-rings.html

13

Re: [Extension release] URL Checker For Posts And Signatures

Can you post the exact full post in code tags here
or email it to me  see my avatar....

and the IPs they've been using?

Edit: found some of their urls and keywords:

[url=http://www.overstocksilver.com/sterling-silver-rings.html]Silver Rings[/url]
[url=http://www.overstocksilver.com/[/url]]Sterling Silver Necklaces[/url]
[url=http://www.overstocksilver.com/diamondczrings.html]CZ Rings[/url]
[url=http://www.overstocksilver.com/tiinstsidicz.html]tiffany key pendant[/url]

* added big_smile

Now we just need our ace Grez to make this code take out those users posting those links... not just warning tongue

14

Re: [Extension release] URL Checker For Posts And Signatures

Ok, so here we got new version of Url checker (v4.0). Its new functionality is to automatically delete users, that try to enter "spamming url" and have less then 5 posts. I've been thinking a lot about that, and I think there's no need to add enable/disable option to administration.

As well, this version contains new .csv (2010-08-24) file, which contains URLs only (for better performance while searching the file).

TODO for version 0.5:
We should figure out how to "canonize" URLs (no spaces, "/" at the end...) and how to rid of duplicates in .csv file neutral
As well, I could try out how would MySQL table instead of .csv perform

Eraversum - scifi browser-based online webgame

15

Re: [Extension release] URL Checker For Posts And Signatures

Great stuff.  Just installed a slightly moded (style - I needed table font smaller) version of it here (packed into the stopforumspam extension).

Couple of features I'd find neat:

  • Data-Forwarding: Collection of IP, Username, Email-Addy, Post messsge from db - then email that info along with the offending URL to the site admin

  • and/or send a copy of the email to me/us (running the url list) so that we can add more URLs to the list - I assume a post will sometimes have new in addition to old URLs)

  • Delete-message: maybe a message alerting to where they can have the URL delisted (if it got on the list by mistake)

  • also echo the admins email addy on the page with banned reason

PS: The list would have already helped delete 3 bad IPs here today - look forward to the first auto delete big_smile Thanks again for the great work!!


Grez wrote:

TODO for version 0.5:
We should figure out how to "canonize" URLs (no spaces, "/" at the end...) and how to rid of duplicates in .csv file neutral
As well, I could try out how would MySQL table instead of .csv perform

I like the idea of importing the list into the MySQL table if its faster and more importantly can be updated via repository!
Duplicates in list: you mean www.badurl.xxx  www.badurl.xxx/xxx or  two times the exact same?

16 (edited by Audiofeeline 2010-08-25 02:05)

Re: [Extension release] URL Checker For Posts And Signatures

Spammers use thousand of domains, fighting against them this way is loosing time I think... The real problem is that forms are not secure without captcha, there's many solution for that... You may look at Akismet or Bad Behavior which are better solutions.  neutral

But this extensions could be interesting for human spamming...  big_smile

17

Re: [Extension release] URL Checker For Posts And Signatures

Audiofeeline wrote:

Spammers use thousand of domains

If the list has 10'000 domains (atm 1'700) and certain domains like ezinearticles (accounting for 1% of spam URLs) are blocked, it should get expensive and time intensive for the spammers, lowering their margins. That in turn makes the business model less attractive.

As always, it's not a golden bullet, but a further option. smile

PS: Which captchas do you use they can't get around?

Re: [Extension release] URL Checker For Posts And Signatures

Using simple captcha image founded on this forum...

19

Re: [Extension release] URL Checker For Posts And Signatures

PS: We just had our first auto-delete here big_smile Some chinese IP wanting bad signature.

Audiofeeline wrote:

Using simple captcha image founded on this forum...

Slavok deleted 20k users that got past that captcha in the last 12 months (or so). And several users including myself have reported that the CAPTCHA on by defaut has been circumvented.
So I don't know if you've changed any other settings (for the captchas or elsewhere) or if your forum is just one not being targeted...

Re: [Extension release] URL Checker For Posts And Signatures

The problem is the captcha himself, too easy to decode. Look at the PhpBB one, it's more complex and really works.
There's another captcha solution using questions edited by the admin like "What is the first letter of PunBB ?"...

21

Re: [Extension release] URL Checker For Posts And Signatures

@Audiofeeline
Thanks, have opened a ticket on trac
http://punbb.informer.com/trac/ticket/349

@Grez

How should we deal with such URLs (one with spaces, the other with backslash) ?

Hello everyoneforum Alex in dat house and I'm a dental tech besides cruising the internet I also like metal detecting
[url=\http://hot guitar.info/\]guitar training video[/url]
[url=\http://someauto. info/\]auto repair videos[/url]
[url=http://hot guitar.info/]guitar videos[/url]
[url=http://someauto. info/]auto review[/url]

22

Re: [Extension release] URL Checker For Posts And Signatures

This really is working quite well. Today 4 users got auto-nuked for wanting to use bad URLs big_smile

http://keydogbb.info/img/urlcheck.png

23

Re: [Extension release] URL Checker For Posts And Signatures

Bug report:

if url tag is followed by domain with backslashes
\www.url.xxx/\

it doesn't get caught.

Re: [Extension release] URL Checker For Posts And Signatures

Extension caught one on my forum today but didn't delete the user.

25

Re: [Extension release] URL Checker For Posts And Signatures

What version are you using?
What does it say in field user for the caught one?