57 replies to this topic

[Mikorist]

Hi folk ,

Some incidental info grabbed off the net:

What I've found to be the most effective, is to make forum #1 a category instead of a normal forum, then place my publically accessible forum inside that category, that way they register, but never get to post anything, because all the spam bots are trying to post into forum #1.

Spammers can hire humans that will register manually at 1000's of forums for them and spam the forums for a small fee. I've seen the sites. Given the increased obscurity of registration processes, it's a growing market - having humans do the spamming for you, which will always defeat the captcha and other registration requirements.

Take a look at this:
http://www.botmaster.net/more1/

Regards,

ispy

Try this captcha

http://www.captcha.ru/en/kcaptcha/

Defeating of some weak CAPTCHAs
http://www.captcha.ru/en/breakings/

kCAPTCHA is meant to be a very strong protected one but requires no special hosting features, only PHP with GD library.

//wave distortion



		for&#40;$x=0;$x<$width;$x++&#41;{

			for&#40;$y=0;$y<$height;$y++&#41;{

				$sx=$x+&#40;sin&#40;$x*$rand1+$rand5&#41;+sin&#40;$y*$rand3+$rand6&#41;&#41;*$rand9-$width/2+$center+1;

				$sy=$y+&#40;sin&#40;$x*$rand2+$rand7&#41;+sin&#40;$y*$rand4+$rand8&#41;&#41;*$rand10;

The script generates random text, writes it to session and outputs image with this algorithm
and writes to it with name $_SESSION['captcha_keystring'] randomly generated text string.
And outputs image with distorted text :



(we have this captcha in our forum)

The XRumer can kiss my as...

[Mikorist]

I'm looking for more optimizations but it seems that the bots got tired for today, should we expect another visit tomorrow?

-----


If it's war they want, it's what they'll get..

Banning bad robots from the site

This is done with a few lines in the .htaccess file. This file contains directives for the web server and are used in this case to redirect all accesses from bad robots to one page, which contains a short explanation why the robot has been banned from the site.

There are two ways to ban a robot, either by banning all accesses from a particular site or by banning all accesses that use a specific id to access the server (most browsers and robots identify themselves whenever they request a page. Internet explorer for example uses Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)", which must be interpreted as "I'm a netscape browser - well, actually I'm just a compatible browser named MSIE 4.01, running on windows 98" (A netscape browser identifies itself with "Mozilla"). In both cases the following lines are used at the beginning of the .htaccess file (note: this works with recent apache web servers, other servers may need other commands):

RewriteEngine on
Options +FollowSymlinks
RewriteBase /

To ban all access from IP numbers 209.133.111.* (this is the imagelock company) use

RewriteCond %{REMOTE_HOST} ^209.133.111..*
RewriteRule ^.*$ X.html [L]

which means: if the remote host has an IP number that starts 209.133.111 rewrite the file name with X.html and stop rewrites.

If you want to ban a particular robot or spider, you need its name (check your access log). To ban the inktomi spider (called Slurp), you can use

RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule ^.*$ X.html [L]

In order to ban several hosts and/or spiders, use

RewriteCond %{REMOTE_HOST} ^209.133.111..* [OR]
RewriteCond %{HTTP_USER_AGENT} Spider [OR]
RewriteCond %{HTTP_USER_AGENT} Slurp
RewriteRule ^.*$ X.html [L]

Note the "[OR]" after each but the last RewriteCond.
The Robot Trap
Three traps are set on this web site:

* Trap to catch robots that ignore the robots.txt file
This site has a special directory that contains only one file. This directory is mentioned in the the robots.txt file and therefore no robot should ever access that specific file.

In order to annoy robots that read that file anyway, it contains special links and commands such that a robot thinks that there are other important files in that directory. Thanks to a special .htaccess file all those other files actually point to the same file. Besides, to load the file takes always at least 20 seconds without using resources on the server.

The .htaccess files looks as follows

RewriteEngine on
Options +FollowSymlinks
RewriteBase /
RewriteRule ^.*\.html /botsv/index.shtml
ErrorDocument 400 /botsv/index.shtml
ErrorDocument 402 /botsv/index.shtml
ErrorDocument 403 /botsv/index.shtml
ErrorDocument 404 /botsv/index.shtml
ErrorDocument 500 /botsv/index.shtml

and the special file uses server side includes, is named index.shtml and the main parts are:

<html><head><title>You are a bad netizen if you are a web bot!</title>
<body><h1><b>You are a bad netizen if you are a web bot!</h1></b>
<!--#config timefmt="%y%j%H%M%S" --> <!-- of date string -->
<!--#exec cmd="sleep 20" --> <!-- make this page sloooow to load -->
To give robots some work here some special links:
these are <a href=a<!--#echo var="DATE_GMT" -->.html>some links</a>
to this <a href=b<!--#echo var="DATE_GMT" -->.html>very page</a>
but with <a href=c<!--#echo var="DATE_GMT" -->.html>different names</a>

The effect is that each robot that hits this page will see new links and request the same page over and over again. Thanks to the 20 second delay the server should not get too busy (unless the robot uses many accesses at the same time, but that would be a very bad robot indeed).

* Trap to catch robots that misuse the robots.txt file
This site has a special directory with the same properties and files as the one above, except that there is no link to it at all. This directory is only mentioned in the the robots.txt file and therefore no robot should ever access that specific file unless it reads the robots.txt file.

Marc has written a program that will automatically ban access to sensitive directories for all clients that access the robots.txt file. I have not tested it though.

* Traps to catch robots that slurp up email addresses
Each of the two files above and an additional one which is plainly visible contain an email address which is generated new for each robot. If that address is ever used, it is trivial to find out who slurped the email address and then block it. To generate email addresses I use

here an email address you better do not use:
<a href=mailto:bot.<!--#echo var="DATE_GMT" -->@ars.net>bot.<!--#echo var="DATE_GMT" -->@ars.net</a>. To make other robots happy as well,

This assumes that the file contains a line as the one above.

Download the traps
If you want to install the traps you can download them here:

http://www.fleiner.c...ts/robotrap.zip

from: fleiner.com/bots/

[ispy]

Hi folks ,

That's a good idea Peter, I've also wiped out my personal message box, less 1247 pm's hogging the DB.

Also, the max PM per member has been reduced to 100.

I would like to clean mine out but would like to back them up on my PC some have good reply info within them is there an easy way of copying the lot in one go, To clean up offline?

Thanks in advance,

ispy

P.S. Tis Okay now I have figured a way! PM Box @ 0%

[dog]

I know it might just be a temporary anti-spam measure, but here's one vote for the fast minimal look

[Arvy]

I know it might just be a temporary anti-spam measure, but here's one vote for the fast minimal look

Make that two votes.

Whatever you've been doing to this server, Nuno, keep doing it. It's working great today. I don't remember ever seeing it this fast.

[pscEx]

Make that two votes.

Whatever you've been doing to this server, Nuno, keep doing it. It's working great today. I don't remember ever seeing it this fast.

And after I found the 'View New Posts' in the upper right, I can live with this solution very well.

Vote #3

Peter

[was_jaclaz]

Make that two votes.

In Italy there is a saying:
[italian]
Non c'è due senza tre
[/italian]

that translates roughly to:

There is no two without a three

.



jaclaz

P.S.: DAMN! peter was faster than me, ok, let's make mine #4

[pscEx]

P.S.: DAMN! peter was faster than me, ok, let's make mine #4

I feel very happy tonight! That's one of the very rare situations that I could beat you!

Peter

[pscEx]

Just to give a proof to Arvy's post #30:

I wrote an reply and stored.
I saw a typo and 'Edit' > Never happened last weeks, I had it within less than one second!

Peter

[ispy]

Hi Voters ,

Still on the fingers of one hand, make that #5 & I personally prefer the colour scheme, one thing though I miss the Thanks button, I've always got a lot of people thank!

R&R,

ispy

[amalux]

Still working great here since Monday morning (post #20) - What a breath of fresh air, it's a pleasure to have the forum working, finally. Please, Nuno, leave it as it is!

[MedEvil]

So that noone can say we're all a bunch of Lemmings, i say, i want the left bar back.

[pscEx]

So that noone can say we're all a bunch of Lemmings, i say, i want the left bar back.

I would prefer to have it, too

But retry for half an hour to see it? ...

Peter

[Brito]

The left bar is good to place informations but mkportal simply cannot continue to be used.

A better alternative is needed.

Some of the mentioned tweaks were added to spoof out these bot farms, thank you for all the tips.

[Arvy]

Fantastic job! Congratulations!

My suggestion regarding an alternative to previous sidebar items would be simple links to whatever you consider essential located elsewhere so that it doesn't have to be loaded with every forum access. I can see no reason why all of the RSS news feeds, for example, should need to appear every time anyone accesses WinBuilder's discussion forums. Surely they could exist on a separate page of their own with approriate links from the home page and/or the top menu, etc.

If some items absolutely must be available for simultaneous viewing, perhaps those particular items could be linked as "on demand" pop-up windows so that, once again, they don't need to be re-loaded repeatedly with every forum access. Or am I being overly simplistic?

[MedEvil]

I don't need any dynamic content in the side bar. But as a convenient place to hold links to where i go most often.
If i could actually choose, which ones those are, would be best.
And i don't see a reason why the sidebar frame would have to be created dynamicly over and over again.
If i would make a change, a new html page will be created for me and from this point on, it's just: serving a static page.

[Brito]

A lot of code that is previously run by php can be replaced by static html and a cron task can be used to regenerate a new page with fresh content at scheduled intervals instead of "every time a new visitor drops by".

This is something that I will apply on the front page for example: http://boot-land.net to remove permanently any need for mkportal.

---

To solve the website integration challenges one can easily use the sdk functions made available by IPBWI --> http://ipbwi.pc-intern.com/

---

The left bar has it's use and reasons to be needed.

One of them for example, is the link exchange agreement between bootdisk.com and our site. It's a popular site in regard to boot disks and they do refer back a lot of visitors but in return we also add a link back to their page to be fair and straight.

I also like to view the online members, it's no fun having to open the main forum page and navigate all the way to the bottom of the page to see who else is online at the same time.

Just a few reasons and in the future I'm sure we'd find better uses without bloating things up and losing performance.

[MedEvil]

Speaking of the start Site Nuno, there was this really annoying thing.
I don't know if it has already been 'fixed', since it's been a while since i used http://www.boot-land.net to come here. But when used this address, a gigantonormous site would start to show up, with 'millions' of posts. What a waste of resources on both ends!

[Galapo]

I don't know if it has already been 'fixed', since it's been a while since i used http://www.boot-land.net to come here.

The left-hand sidebar is still there on that page!

[Arvy]

The left bar has it's use and reasons to be needed.

One of them for example, is the link exchange agreement between bootdisk.com and our site. It's a popular site in regard to boot disks and they do refer back a lot of visitors but in return we also add a link back to their page to be fair and straight.

Perhaps we're saying the same thing in different ways, but, just to be clear, my suggestion wasn't necessarily intended to eliminate ANY content. Nor was it based on any personal aesthetic preference. It's just a question of optimal performance placement within the site as a whole.

Simple static links should not impose any significant problem with page re-loading regardless of their location. On the other hand, it would seem logical to keep the most highly dynamic part of the entire site as clear of additional dynamic loads as possible. So, assuming that overall site performance is to be a major consideration, the placement of any such added burdens on the forums pages should probably be avoided except where there's no alternative. I don't know if that dynamic issue applies to the bootdisk.com agreement, but, if so, would they object to placement on this site's home page or some other prominent place that doesn't incur the same heavy loading and re-loading demands as the forums do?

Of course, satisfying demands for everything to be visible on every page of the entire site does get a bit tricky, especially if they want it all to be at the top.

[Mikorist]

Some of the mentioned tweaks were added to spoof out these bot farms, thank you for all the tips.

Remove All Admin Logs

TRUNCATE TABLE `ibf_admin_logs`

Remove All Moderator Logs

TRUNCATE TABLE `ibf_moderator_logs`

Remove All Email Logs

TRUNCATE TABLE `ibf_email_logs`

Remove All Email Error Logs

TRUNCATE TABLE `ibf_mail_error_logs`

Remove All Profile Comments

TRUNCATE TABLE `ibf_profile_comments`

Remove All Friends From All Users

TRUNCATE TABLE `ibf_profile_friends`

Remove All Profile Ratings

TRUNCATE TABLE  `ibf_profile_ratings`

Remove All Spider Logs

TRUNCATE TABLE`ibf_spider_logs`

Remove All Task Logs

TRUNCATE TABLE `ibf_task_logs`

Remove All Warn Logs

TRUNCATE TABLE `ibf_warn_logs`

This deletes all Private Messages stored for all members.

TRUNCATE TABLE `ibf_message_text`

TRUNCATE TABLE `ibf_message_topics`

[pscEx]

Currently I'm testing beta 4 u
And that brings a lot of logs to be viewed in the browser.
The last log needed 40 seconds until the end was visible.
I repeated and tried to scroll to the always actual end.

And I could see, how every icon was downloaded. No caching on my machine ?? !!

That means: A standard LiveXP log demands around 13000 downloads from the bootland server.

I am sure that one of the first tasks of beta 6 is to store the icons locally in %BaseDir%\images


Peter

EDIT: I made a test with locally stored images. It's not really faster! So my theory is propably wrong!

Edited by psc, 11 September 2008 - 04:52 PM.

[john3voltas]

I have always been a friend of minimalist.
Even my preferred php forum script is FluxBB which is a fork of PunBB and both of them are primarily slick and minimal.
So I'd add a vote to the forum without left bar even though Nuno will have to find a way to put up a bootdisk.com link here in the forum.

[Brito]

We're stuck on IPB software for the forum but let's see if we can make the forum run faster, mkportal was adding to much bloat and after disabling it things have been working a bit better.

[Arvy]

This is a case where my sympathies tend toward the "minimalist" side. Nevertheless, if you feel that some kind of sidebar additions are essential, perhaps the IPB Widgets add-on could achieve most, if not all, of what you want there without the incurring the very heavy loading penalty of mkportal. If I recall our earlier experimentation correctly, the IPB Widget's display capabilities aren't necessarily limited to the board's own built-in features alone.

Just a passing thought that might possibly be helpful in achieving the end result that you want.