Jump to content











Photo
* * * * * 1 votes

server died today afternoon


  • Please log in to reply
51 replies to this topic

#26 Dramastic

Dramastic

    Member

  • .script developer
  • 55 posts
  •  
    United States

Posted 27 October 2008 - 10:52 PM

If it is the LiveXP files being attacked, then may I suggest possibly implementing some sort of 'handshake' between WinBuilder and the download site that the attackers don't know? This could limit downloads to WinBuilder only.

Even something simple could make a difference. For example if the downloads are simply protected by an FTP password, and WinBuilder told to find the password say on a webpage on the same server using http? Then you can manually or automatically change the password any time there is an issue. Attackers would need to learn how this procedure works before they could cause further problems. Chances are they are not that interested.

Dramastic

#27 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 28 October 2008 - 12:17 AM

Dramastic, to get around your security system, would take a simple sniffer, WB and a few seconds time.
Not much of an obstacle to discourage an attacker.

If you wanna do it right, have the server send a request, that WB has to process in the correct way and send back, to gain access.

:cheers:

#28 Dramastic

Dramastic

    Member

  • .script developer
  • 55 posts
  •  
    United States

Posted 28 October 2008 - 06:29 AM

small_cannon_fly.jpg

Dramastic, to get around your security system, would take a simple sniffer, WB and a few seconds time.
Not much of an obstacle to discourage an attacker.


That is correct. Nevertheless they may not bother. It depends on the kind of attacker.. most times it is enough to build your fence a little taller than your neighbors..

Attackers would need to learn how this procedure works before they could cause further problems. Chances are they are not that interested.


If you wanna do it right, have the server send a request, that WB has to process in the correct way and send back, to gain access.


Definitely an improvement to my suggestion - unfortunately, as long as the attacker has the WinBuilder exe code to examine & mimic, I think this will have little additional effect if the attackers are as enthusiastic as you imagine.

If it is to be 'done right', then a few approaches come to my mind at the moment. I am sure there are many others. One approach would be to require login/password for each user of Winbuilder - many of us already have a forum account that could be used - but there may go anonymous usage for the rest. The other would be Captcha - which I am pretty sure won't be implemented by itself because of the level of annoyance. Of course the two approaches can be combined, allowing users the option of either or. The last approach that comes to mind might be to use https/ssl. It wouldn't stop an attacker as such, but it could help better identify them so they can be blocked.

But it is all very theoretical. Who is going to program it? The less work to implement, the more likely it will be done. If existing standard server software is used, there will be less work to do. If the attacks persist, then I agree it may be necessary to take additional steps to scale the response to the situation.

One approach to becoming victorious in a serious attack situation is to create as many obstacles for the opponent as you can with as little effort on your part as possible and requiring as much effort as possible on theirs. Sooner or later they will become exhausted. My suggestion was merely intended as a first blow. Scale as needed.

Dramastic

#29 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 28 October 2008 - 01:21 PM

Some counter measures were added to liveXP server.

I won't post extensive details due to security reasons about the performed changes but it should allow the normal functioning of winbuilder.exe and related sites. Will check the statistics after being active for 24 hours.

There are still a few other weak points - I notice that winbuilder.net itself is also being targeted and this brings down the MySQL server does anyone know any php script capable of blocking an IP after n page loads?

Thanks.

:cheers:

#30 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 28 October 2008 - 02:20 PM

Definitely an improvement to my suggestion - unfortunately, as long as the attacker has the WinBuilder exe code to examine & mimic, I think this will have little additional effect if the attackers are as enthusiastic as you imagine.

The idea was to raise the amount of required work to discourage the attacker.
To circumvent the above, there are no tools, one has to be able to read and uderstand assembler.
Which would already rule any script kiddies and lamos out.

:cheers:

#31 Dramastic

Dramastic

    Member

  • .script developer
  • 55 posts
  •  
    United States

Posted 28 October 2008 - 03:49 PM

Have not looked at it, but perhaps you could tweak this to get what you want..

http://www.hotscript...iled/78344.html

#32 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 28 October 2008 - 05:38 PM

Have not looked at it, but perhaps you could tweak this to get what you want... http://www.hotscript...iled/78344.html


Well it's certainly a heck of a lot cheaper than a Cisco Systems hardware solution. :cheers: And the "timeout" concept is fundmentally sound as it only relies on IP addresses for that limited period. Unfortunately, it also relies on redirection using a PHP script which still leaves the actual download file open to direct access. Nevertheless, it could be worth a try as one very simple way to discourage some of the less sophisticated script kiddies.

Personally, I'm still inclined to think that the ultimate solution would be an intergrated database download management system for WinBuilder, but Nuno hates databases in general.

#33 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 28 October 2008 - 05:47 PM

Hi Arvy, would you mind modifying the script to only allow n page loads?

Or perhaps even create a new script for this task? :cheers:

I've looked it up and it's using a text file on the server to log and look up the IP's maybe a mySQL solution would come in handier.

I remember that you are experienced with e107 - what we need here is to limit the number of page loads done over at the winbuilder.net and CMS plugin for this task would be really good (don't know if one exists already, maybe I try to find it later).

----

Say for example only allowing 100 page loads per IP on a single day ought be enough to keep the slurpers away from the site.

On boot-land.net it would be nice to limit members with 0 posts to only 100 page loads, this could be possible be improved by adding a visual counter saying ("30 page loads left for today").

--

Just a bunch of random ideas, I lack the time to properly work on them for the moment.

:cheers:

#34 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 28 October 2008 - 05:56 PM

I'll take a look at the possibilities and get back to you about options. But you'll then have to make a firm decision about some particular route to follow and stick with it. My "charging madly off in all directions" would just be a waste of time for both of us.

#35 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 28 October 2008 - 06:13 PM

Let's look on all possible perspectives as necessary.

:cheers:

#36 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 28 October 2008 - 06:34 PM

Quick question, with just 100 page loads per day, can a project like LiveXp still be downloaded?

:cheers:

#37 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 28 October 2008 - 07:06 PM

As wikipedia wouldn't say: "This is an ambiguation".. :cheers:

WinBuilder's download center is left unrestricted as before, what would be restricted is the abuse of winbuilder.net where spam bots might try to crawl several times to spam or download available files.

:cheers:

#38 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 28 October 2008 - 07:10 PM

Based on previous efforts to help, the "all perpectives" perpective is the one that I fear most. Some perspectives make good sense and some make no sense whatever in the context of existing realities -- such as the whole issue of ownerships and permissions under the virtual server's web root, for example. We've spoken privately about that in the past and, unless that kind of fundamental issue is dealt with, putting extra locks on the doors to a rooom with no walls is utterly futile. As another example, I'm not even sure if the WinBuilder virtual server operates with suexec or uses the Apache "nobody" user for PHP scripts. Do you know?

Likewise, as in the issue raised by MedEvil, one needs to have a clear and precise grasp of specific objectives. It's one thing to address a particular issue like "bot wars" and their consequences. It's something else entirely to address vague notions about relative privileges of forum members, non-members, etc., etc. In fact, the entire issue of the relationships between these forums and the broader Boot-Land and WinBuilder contextual and functional environments is by no means clear -- at least not to me. How does the level of anyone's forum discussions take on a governing relationship with their WinBuilder project downloading, for example? Are we really seeking to exclude or limit newcomers to the WinBuilder PE experience who may not even need or want help from this source? Why?

#39 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 28 October 2008 - 08:08 PM

It is nearly impossible to fully evaluate and draw precise guidelines regarding what needs to be done since it seems that once a weak point is identified and solved there is quickly another one crumbling up the available resources that need immediate attention.

I'll be working to solve the issues that are currently menacing the server as they appear.

:cheers:

#40 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 28 October 2008 - 08:33 PM

I know, hot patching is the favorite kind of addressing problems worldwide.
But a bit of planning for a more general solution, goes a long way in winning the fight. :cheers:

:cheers:

#41 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 28 October 2008 - 08:38 PM

I know that you are neither stupid nor obtuse, Nuno. You know very well that I wasn't asking you to provide "precise guidelines regarding what needs to be done." However, it is impossible to help anyone without a very clear mutual understanding about the objectives and about the environmental framework within which those objectives are to be accomplished. As I said, many aspects of both are quite unclear at present and the particular questions that I did raise remain unanswered.

Anyhow, you go ahead with whatever you think may provide a solution in your own estimation. If you encounter any specific issues along the way, I'll help to whatever extent I can.

#42 Dramastic

Dramastic

    Member

  • .script developer
  • 55 posts
  •  
    United States

Posted 28 October 2008 - 09:36 PM

The fundamental issue here appears to be lack of any information about the attacker(s). Without this it will be difficult to give Arvy any clear direction or instruction. If you can even identify their motivation, you are likely to be able to diffuse the whole situation. It could be anything from a broken bot to someone brute-forcing a password.

Nuno, is there any way that you can position yourself better to obtain more information about the attackers? For example, could you redirect the domain name to point to a new IP address that simply forwards all requests to your current IP & host? If the new IP host is willing to share more traffic details than your current host, you may be able to come up with a better strategy and give Arvy a fighting chance to have his efforts have an impact.

I don't know what your relation is with your hosting provider. Perhaps they are friends or relatives or you somehow feel you cannot request their assistance. I think most customers in your situation would be complaining to them by now. They should either provide you means to defend yourself or do the defending themselves.

Have you considered setting up a mirror site? If its the WinBuilder files (which don't need to be up-to-date-to-the-second) as opposed to the forums, drop me a note. I am sure there are plenty of people here willing to provide some server disk space to help you overcome the attacks.

Dramastic

#43 Arvy

Arvy

    Frequent Member

  • Developer
  • 430 posts
  • Location:Canada, Parry Sound
  • Interests:IT, Outdoors, Horses
  •  
    Canada

Posted 28 October 2008 - 11:54 PM

In fact complete Apache raw access logs are being maintained and are available for the virtual server, but they're not being analysed or used at present. See this discussion thread.

I could probably live without all those details, but not without a clearly defined and mutually agreed objective -- i.e., are we actually trying to stop "bots", or are we trying to tie WinBuilder project downloading into some kind of forum membership criteria, or a combination, or what? And are we prepared to close other exisiting server vulnerabilites to achieve whatever the goal is, or not? Etc., etc. It all seems quite muddy from my perspective.

In any case, it seems to be a moot point as Nuno has apparently decided to pursue his own course of action.

#44 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 29 October 2008 - 12:58 AM

I think, since what's causing the problem is still in the dark, all suggestions for fixes are just shots into the dark.
I think, what Nuno want's is, to give everyone unregulated access to post and download.
On the other hand is the problem, that there must a way be found to stop or reduce the the load, that causes the server to be, far too often not reachable.

At the moment, the general opinion is, that this load is caused by misuse or attackers. But if that is really so?

:cheers:

#45 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 29 October 2008 - 09:21 AM

I'll risk to be accused of thread hijacking :cheers: and I will tell you a couple of stories, that may be of help to Nuno in this moment of his life. :cheers:

When I was a little child, say 6 months old, one day I started crying with no apparents reasons.

After having tried all "normal" remedies to no avail, my mother called my grandmother ( who had a number of sons and daughter and was very expert in managing babies) for help.

The first think my grandmother did was shouting:

GET HIM NAKED, COMPLETELY NAKED!


Sure enough, the reason of my crying was found to be an elastic band in the sleeve of my shirt that was too tight.

I was always told this episode of my childhood as an example of how experience and a rational approach may help in solve problems.

Several years later my cousin had a baby, who one day started crying with no apparent reason, I was at her home at the time and first thing I did was (you guess :cheers:):

GET HIM NAKED, COMPLETELY NAKED!


And soon we found that a looped thread in the pijamas' trouser was almost cutting through the tender flesh of the baby.

I adopted the same approach when examining the malfunctioning of PC's:

REMOVE ANYTHING, I mean ANYTHING but keyboard and video!

and of almost anything:

Disassemble up to the bare minimum CORE and rebuild step by step

with generally good results.

jaclaz

#46 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 29 October 2008 - 11:09 AM

Good story, I think it is quite appropriate.

I use this methodology myself when trying to figure why some code portion isn't working as intended.

---

At the moment I am already trying to achieving this by reducing to a minimum the number of addons and doing a cleanup that was needed since some time ago.

---

MedEvil, I could learn a lot from your communication skills - I think you summarized our current situation very well.

Checking the LiveXP bandwidth stats I see that my attempts yesterday didn't had much impact on the traffic slurping.

The possibility to mirror this specific server is an option but how many people would be available to support >60Gb of downloads per day?

I also have a lot of difficulty interpreting the awstat logs if anyone around here has experience catching these bots I'd surely appreciate a hand.

:cheers:

#47 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 29 October 2008 - 11:49 AM

Maybe we can start by checking which resource is actually the one running low.

Every Server has 3 major weak spots.
Processing power = CPU/RAM
Network bandwidth
Bus bandwidth of the internal storage system

Next step would be to figure out, what causes this shortage. Is it a misconfiguration or an external thing.

If it is an external thing, the hard part begins. Getting a grip on what that external thing is.

Once those points are cleared up, a strategy can be plotted, how to 'protect' the resource better without affecting the normal operations too much.

:cheers:


PS: Since the new WB-Betas seem to have trouble with the downloading, yet everyone seems to already use them, can it be that the WB-Betas causing this havoc?
I mean, think about it, LiveXP has the most troubles and LiveXP is the one that even requires a Beta!

#48 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 29 October 2008 - 12:53 PM

@all
Refreshing the error page brings again only a refreshed error page! :cheers:

@jaclaz
Nice story!
Next time you scream at me, for no reason, i shout too:

GET HIM NAKED, COMPLETELY NAKED!

:cheers:

:cheers:

#49 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 29 October 2008 - 01:15 PM

@jaclaz
Nice story!
Next time you scream at me, for no reason, i shout too:

:cheers:

:cheers:


Hmm :cheers:, I doubt you will like the view. :cheers:

jaclaz

#50 Nuno Brito

Nuno Brito

    Platinum Member

  • .script developer
  • 10545 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 29 October 2008 - 01:20 PM

Why am I laughing?

This was meant to be a serious talk about servers getting killed this afternoon and stuff. Please behave.. :cheers:




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users