51 replies to this topic

[billonious]

server overdrive!

[Brito]

I've ran a detailed check and this time it is no spam bot action trying to attack the forum.

The MySQL forum database was reduced by -80% from over 1Gb to less than 200Mb and the software was optimized to support a very high demand so I thought that all of these reactions on the server were strange until I checked the bandwidth stats for all our server domains that clearly displayed an anomaly.

http://livexp.boot-land.net has used over 500Gb of bandwidth over the last 9 days, this should total 1,5~2Tb worth of data that are creeping up the CPU/RAM resources needed by Apache to supply the demanded files.



It's not easy to download such amount of data in such short time. Comparing to last month - liveXP was using around 10Gb per day and now was already going on 70Gb used today.

--------

I'm working to solve this matter.

[Brito]

Looking on the specific subdomain logs for the last three days it shows 42065 downloads of updates.ini

This means around 584 project downloads per hour or nine requests per minute..

I don't know if most of them come from the same IP or not, the stats software (analog 6.0) doesn't account these values and it is difficult to block access to the bots or scripts doing this sort of things.

[Dramastic]

Is this at all related to that?
Error @ homepage:


[codebox]<br /> <b>Warning</b>: require(./mkportal/conf_mk.php) [<a href=&#39;function.require&#39;>function.require</a>]: failed to open stream: No such file or directory in <b>/home/.fluke/nuno_brito/boot-land.net/index.php</b> on line <b>28</b><br /> <br /> <b>Fatal error</b>: require() [<a href=&#39;function.require&#39;>function.require</a>]: Failed opening required &#39;./mkportal/conf_mk.php&#39; (include_path=&#39;.:/usr/local/php5/lib/php:/usr/local/lib/php&#39;) in <b>/home/.fluke/nuno_brito/boot-land.net/index.php</b> on line <b>28</b><br /> [/codebox] [codebox] Warning: require(./mkportal/conf_mk.php) [function.require]: failed to open stream: No such file or directory in /home/.fluke/nuno_brito/boot-land.net/index.php on line 28 Fatal error: require() [function.require]: Failed opening required &#39;./mkportal/conf_mk.php&#39; (include_path=&#39;.:/usr/local/php5/lib/php:/usr/local/lib/php&#39;) in /home/.fluke/nuno_brito/boot-land.net/index.php on line 28

[/codebox]

[MedEvil]

Assuming that those is bot created traffic, how about disabling index.html for the projects to test if it really is?
Never saw any use in that feature anyhow.

[Dramastic]

Not entirely sure as I get another error page when I try to retrieve it, but seems you may not have a robots.txt file either. Of course if I can't get your robots.txt, probably neither can the bots that actually heed it..

http://www.robotstxt.org/


Dramastic

[pscEx]

Looking on the specific subdomain logs for the last three days it shows 42065 downloads of updates.ini

This means around 584 project downloads per hour or nine requests per minute..

I don't know if most of them come from the same IP or not, the stats software (analog 6.0) doesn't account these values and it is difficult to block access to the bots or scripts doing this sort of things.

To be sure that it is nor introduced by a new WinBuilder version, I made a test under the debugger.

Result:

updates.ini is only downloaded when the user enters the download page.

Peter

[pscEx]

Maybe it helps to report an issue I just got:

I did a download of a 'complete' LiveXP.

It started using my full bandwith, showing with small files just a flicker, and with bigger files the megabytes changing.

Then it became slower and slower, showing kilobytes changing.
Anywhen the 'downloaded' number did not change for several seconds and than started again changing slowly.

After finishing the file, download stopped.

I restarted: It run with the old speed and continued until all files were download.

The stop between is an issue Ill look for in the source.

But the server's hehaviour is strange.

Peter

[Brito]

It's not a "regular" bot to be controlled by robots.txt

We can however limit it's access permission on .htaccess if we know which IP it is using.

The LiveXP location is perfectly indexed by regular crawler bots and these "things" are different because they are massively downloading all files or at least one single big sized file simultaneously until the server gets in overload.

Last night it was necessary to reboot the server several times to get things working again, this seemed like the only remedy.

The subdomains became a weak point because we never had any issues before, guess it will be necessary to remove direct link download from livexp.boot-land.net and only allow from the respective index.html or from a wb client if this sort of thing continues.

-----

I also thought it was related to winbuilder and even mentioned this to Peter about two days ago when testing the newer beta but now I'm convinced that this odd behavior came from the server.

-----

There are at least three ways to consume such level of bandwidth so quickly.

#1 someone scripts several wb instances to continuously loop download everything from the available server.
#2 another server online loops several instances of webget to download big sized files
#3 the number of wb users increased beyond expected results too quickly (being mentioned on a magazine or popular website for example)

One quick solution would be disabling the liveXP server by renaming updates.ini and we'd quickly discover if methods #1 or #3 mentioned above are valid or not. If it's #2 it can be later today.

[pscEx]

It's not a "regular" bot to be controlled by robots.txt
We can however limit it's access permission on .htaccess if we know which IP it is using.

Yesterday I saw archive.orgs as user, and they host coplete web site copies!

Maybe you try to exclude their IP

Peter

[MedEvil]

Today this darn server is really getting on my nerves.
I'm used to topics that keep the 'new post mark' no matter how often i visit them.
But today the marks appear and disapear without any methode to it.
Some 'old' topics get the mark all of a sudden again, while a new one does not!

[Brito]

Peter, our server uses little above 15Gb of overall disk space (and most of it's use is due to my personal photo collection).

Wouldn't make much sense for archive.org to download 40~70Gb per day.

----

MedEvil, we're working on this matter.

As you requested, the default list of available web servers has been modified. If LiveXP continues to be targeted I'll temporarily disable it and talk with Galapo to move onto an alternative location until things get back to normal.

This is mostly affecting the forums, I'm also in conversations with R1Soft as they are interested in sponsoring a dedicated web server to be used by our community for whatever necessary and I wouldn't mind using it as external location to balance the server load. More details should be posted as soon as the dust settles.

[Galapo]

to me it would seem that the excessive download from livexp is automated. Sure, I posted a 170mb archive on WinBuilder.net, but that has only been downloaded 416 times since 13 October. As far I am aware, no direct links to the archive have been provided, so a normal user needs to go through the web page first and hit the download button.

Regards,
Galapo.

[amalux]

Strange message when going to http://www.boot-land.net/ (from Google search)

sub-links ok but main heading gives this message

also, I assume you know you have to login everytime you leave boot-land and come back, even a second later.

[Dramastic]

Nuno, I guess that is Apache? With which I have no experience - but if your Servers on a Windows box with GUI access, try this:

http://www.freshsoftware.com/xns/

or just use "netstat" from DOS prompt..


Dramastic

[Brito]

Amalux:

On http://boot-land.net it's "normal" to see the error message.

I've removed completely mkPortal to test if it was being targeted or not, I'll be updating the portal page with a static HTML page to avoid slurp of resources.

------


Galapo: Don't worry. It's not liveXP's fault, this is likely an automated script meant to achieve such levels of bandwidth usage.

The author might as well have chosen other single big sized files like the UXP project with 600 Mb (uxp.zip).

----

Dramastic:

Our server is a debian machine with apache located on a Virtual Private Server hosting to which I have no root access nor X windows available.

This should hopefully change soon enough.

[MedEvil]

Ehm, what's that? I all of a sudden see the IP adresses of posters above their posts?
http://www.boot-land...amp;#entry48011
Is this a new feature for the 'admins' or does everyone see these?

[Dramastic]

Virtual Private Server hosting

Perhaps then ask the provider for some insight & assistance in the matter. I am sure they are not interested in using that amount of bandwidth unnecessarily either....

Dramastic

[MedEvil]

Today is a fine day to surf Boot-land!
Boot-land works so fine, makes me long for the speed my old 14,4 modem gave me!

Without attacking anyone, but it seems the more Boot-Land gets optimized, the worst the performance gets.

[Brito]

The board is optimized and now the troubles are occurring on the sub domains..

[pscEx]

The board is optimized and now the troubles are occurring on the sub domains..

If this can help:

I can temporarilys delete all my boot-land.net subdomain pages and transfer them to my own server (what since a while I'm already already doing because of FTP troubles. Currently I try to make a copy to boot-land.net whenever FTP is possible)

Peter

[Brito]

It shouldn't be required.

What we need is good counter measures to prevent the server from being attacked.

At the moment they only seem to be picking on LiveXP. I'll try to see if during the next week this can be solved.

[Arvy]

Offhand, I can think of only two ways that downloading by "bots" could be blocked without adversely affecting ordinary users: by IP address blocks and/or by quotas based on timing or other factors.

Since you say that your current log analyses don't provide the required IP address information (?!), that would seem to leave only one other option. IP address blocking is seldom very effective in any case. Since DHCP and other allocation methods re-assign addresses frequently, you'd be as likely to block the innocent as the guilty.

[pscEx]

Something really seems to go wrong!
I just did an FTP upload of around 10 MB 'nightly' w/o any troubles

The speed has been constantly at 669.?? kbps. Only the ?? changed.
That is using full bandwidth of my connection.

Who changed what?

Peter

[Galapo]

I generally have always gotten full bandwidth on ftp upload to bootland: usually between 6.0 and 6.6kbps.

Regards,
Galapo.