16 replies to this topic

[Nuno Brito]

Well.. unfortunately, the server where winbuilder.net resided got (supposedly) hacked 10 days ago.

There was a complaint that was made to the hosting company where winbuilder.net was hosted and this was followed with a swift shutdown and blocking of the server. Supposedly, we were serving a malicious page used for phishing at my own personal site (nunobrito.eu) that was also hosted at the server. (if I wanted to publish a phishing page, I wouldn't surely put on my own page but what can I say.. )

I've been trying to bring back the server online or at the very least get our files back but they only provide access to the server under the condition that it is first formatted and all files deleted.

Needless to say that we no longer hold maintain a hosting contract with this company.

There was a backup made about two months ago of all the subdomains of both boot-land.net and winbuilder.net that were hosted at the server. I've restored this backup and put them back online with the help of Mikorist.

boot-land.blogspot was also shutdown last week due to a complaint of spam-behavior and we've been also receiving abuse/spam warnings from the hosting company where boot-land.net is hosted but luckily they gave us a chance of providing a proof that there is no veracity in these complaints and boot-land.net was not put in offline condition.

--------------

With all these issues, some security measures need to be taken to tighten our defenses.

Given the fact that the portal where both nunobrito.eu and winbuilder.net were running can be hacked, I've moved my personal page to the blog at http://nunobrito1981.blogspot.com and winbuilder.net is now pointing to http://www.boot-land...hp?showforum=22

I've done a very modest introduction to winbuilder at the root of the winbuilder forum here on boot land. Will try to improve the current state of things as time moves on but at least winbuilder.net will no longer point to a blank page and a download link to the latest version is currently available.

winbuilder project servers should be fully working as before but one drawback from this change is FTP access.

At the moment we are avoiding the grants of FTP accounts to the server as their respective passwords can be brute-forced. (even SSH is under attempts of password guessing)

One idea is using dropbox to share some folders with trusted developers that need to perform changes on their webspace although we are still working to provide a better long term solution that can be self-managed.

-----------------


So, my apologies for the current state of operations. Should take some months to get back to normality but at least we're still here.

[Galapo]

Hi Nuno,

Thanks for all your hard work in trying to get the situation back to normal.

I wonder why a complain isn't allowed to be defended but they just shutdown your server without a chance to retrieve files. That's quite nasty!

So what's the way forward? If FTP is no longer an option, how can we upload new scripts and project files to, say, livexp.boot-land.net etc?

Thanks again,
Galapo.

[homes32]

keep up the good work! I know it can be frustrating!

as far as brute forcing is concerned I have had wonderful success using fail2ban on my linux boxes. very easy to setup and configure.

[Nuno Brito]

So what's the way forward? If FTP is no longer an option, how can we upload new scripts and project files to, say, livexp.boot-land.net etc?

We'll temporarily be using dropbox as an alternative. Just install it on your machine and then send me by private message the email address that you use on their service so that I can share the folder with you.

as far as brute forcing is concerned I have had wonderful success using fail2ban on my linux boxes. very easy to setup and configure.

Good suggestion, wish I'd installed it some months ago..

I'm quitting the use of FTP also because it has become difficult to manage. We have many users and every time we need to pack our stuff and move to another server (this is the third server in 4 years) then it is surely a bothersome to replicate all the accounts and link them to the right folders.

We also have little control on what is uploaded, even trusting on the good faith by developers - malicious people will always find flaws on the hosted pages and exploit them without need for FTP. Would be better if instead of single pages, single topics on the forum could be created as it is surely more robust to these attacks.

--------

My goal for the next months is to create our own way of uploading files to sub domains using the forum structure.

The first part was creating a plugin that allowed a self-service-style sub domain feature: http://community.myb...read-76106.html

The second part is dealing with the file upload.

- Replace the current forum attachment system for future uploads
- Should not interfere with previous attachments
- Users should be able to manage files with folders
- Keep track of weekly statistics (download counter)
- Allow users to set permissions on folders/files - public, private, registered members only
- Allow setting a quota (default of 5 Mb)
- Be fit for use of maintaining wb projects
- Prevent abusive download (leech/DDoS attempts)
- Allow directory browsing
- File rating by visitors
- Allow sharing folders with other members
- Allow interface with external tools to maintain files on server

So, I'm using sqlite to create a database specific for each user and planning to use it for storing log details, settings and the files themselves while keeping the forum DB lightweight.

I don't like raising expectations but this is the solution that I've been planning over the past two weeks to mitigate what we've been passing through right now. The goal is to ensure that files are kept portable and manageable. If and when we need to move elsewhere, we'll just bring all the files and sub domains along with the forum.

If someone is no longer around to manage a project (as seen on Joshua and nightMan), we can assign trusted members to maintain the project folders in good shape. If someone is hosting something on boot land that shouldn't be there, users should have the conditions to also report this to a moderator.

[Wonko the Sane]

Hmmm.

Besides all the nice bells and whistles, wouldn't i be advisable to have *somehow* a read-only MIRROR on another server (automatically and differentially updated every, say, three days) in order to avoid this kind of issues?

About the supposed phishing page, maybe it is not that much supposed, until you have ANY form of ads/publicity "automagically" linked to/served to by the outside WITHOUT manual (human) approval you will be subject to this, one way or the other.


Wonko

[Nuno Brito]

About the supposed phishing page, maybe it is not that much supposed, until you have ANY form of ads/publicity "automagically" linked to/served to by the outside WITHOUT manual (human) approval you will be subject to this, one way or the other.

The complaint mentioned a very specific link inside my personal site. This link did not belong to the structure of the site and were separated html pages, my guess is that they were injected with resort to some exploit assuming that someone has indeed hacked the site and they existed.

There are many exploits that I have (and you) identified through ads and links on the forum.

This is actually one of the reasons why we currently only use adsense from google and are moving the board software from IPB to MyBB. Albeit you don't like the automagic features of the forum, they are not the reason to blame for the current infection of winbuilder.net and nunobrito.eu or the subdomains as seen some time ago and the forum issues are being dealt as time provides.

Besides all the nice bells and whistles, wouldn't i be advisable to have *somehow* a read-only MIRROR on another server (automatically and differentially updated every, say, three days) in order to avoid this kind of issues?

Do you remember when 8 months ago in January we were plagued with all the DDoS attempts?

If the hosting company had locked down the server back then, we'd be left without any files, subdomains nor even an updated database backup of the forums.

Since then, we got a second server to split the daily load, replaced apache by lighttpd to make the forum more responsive and also automated the backup procedures onto dropbox as the backup box in case all things go wrong. We also improved the robustness to successfully resist DDoS.

This is a work in progress, sub domains take 16Gb of size and at most I perform a backup from one server to another but this has been a manual process and takes a very long while to complete (besides the obvious bandwidth risks), dropbox is limited to 10Gb on the free version and priority was given to the forum backups.


So, we are cleaning up the sub domain space to slim down this huge size, removing attack vectors such as FTP accounts, avoiding single pages with PHP that can be hijacked, limiting the access to MySQL and disallowing other services to be added (wiki, CMS, etc).

We also ensure that we keep a secondary server at hand to replace the primary one if it fails.

So, a lot has already been done since the beginning of the year. Still a lot more work ahead and the few bells and whistles we already have were already enough to prevent a total loss.

Of course that we keep learning and moving forward to prevent these events from reoccurring.

[Wonko the Sane]

This is a work in progress, sub domains take 16Gb of size and at most I perform a backup from one server to another but this has been a manual process and takes a very long while to complete (besides the obvious bandwidth risks), dropbox is limited to 10Gb on the free version and priority was given to the forum backups.

...and most of this I presume because of the inane amount of:

• duplicated files
• unneededly embedding apps in .scripts
• dormient or abandoned projects

(that's why I was talking about "differential" backups, I have no access to the statistics, but I could bet that a lot of the traffic and of the size is due to something that has not changed lately )


Wonko

[Nuno Brito]

...and most of this I presume because of the inane amount of:

* duplicated files
* unneededly embedding apps in .scripts
* dormient or abandoned projects

duplicated files - Not sure, might be. Ask developers to be nice.

unneededly embedding apps in .scripts - I disagree, there is the need of embedding files:
- to ensure that all files are ready for deploy (if license agrees)
- save time gathering files to end user
- avoid version mismatch conflicts

These would be reasons that you'd promptly identify if you developed .script files or even used them but that doesn't deter you from providing an opinion does it?

dormient or abandoned projects - As far as possible, we will avoid deleting inactive projects.

[Mikorist]

So, we are cleaning up the sub domain space to slim down this huge size, removing attack vectors such as FTP accounts, avoiding single pages with PHP that can be hijacked, limiting the access to MySQL and disallowing other services to be added (wiki, CMS, etc).

We also ensure that we keep a secondary server at hand to replace the primary one if it fails.

We have large-scale of daily attacks on server - every five minutes someone trying to break the server here
through all of the protocols and ports...it is simply amazing how much someone wants to make this server
stops working and disappear all - Nunos effort and work

sub domains take 16Gb of size....

plus also :

Backup size:
34G total ---> for sites 33G total and databases 994M total......

read-only mirror on another server is great idea but we must have at least 50GB of space and
very fast link there also......

[Wonko the Sane]

duplicated files - Not sure, might be. Ask developers to be nice.

You have access to the server, I haven't.
Count how many instances (just as an example) there are of:

• 7za.exe
• mkisofs.exe
• qemu.exe

This is of course NOT a "developer originated" problem, it's an architectural issue, as well as the apparent inability of avoiding the re-re-re-re-re-re-download of the same §#@çing file over and over.

The advice, very OFTEN given on the board in reply to problems is, "start from scratch", re-download the whole project!

This ADDS to the entropy....

unneededly embedding apps in .scripts - I disagree, there is the need of embedding files:
- to ensure that all files are ready for deploy (if license agrees)
- save time gathering files to end user
- avoid version mismatch conflicts

So, if there are, say, 15 different projects, all embedding, say, Opera, there will be the size of the embedded Opera x 15.

As said now for the umpth time, embedding the apps may be OK for very small thingies, but not for big chunks of software, if there were infinite resources, that would be OK , but if these mindlessly embedded apps (and their downloads) should be even a minimal cause of problems to the "whole" project, and a cause for more inconveniences to members and various whinings around.


In a perfect word, one would have unlimited resources, but we are not in a perfect word, and thus reducing as much as possible the needed resources or better using the ones available is only inevitable.

These would be reasons that you'd promptly identify if you developed .script files or even used them but that doesn't deter you from providing an opinion does it?

Don't underestimate too much the knowledge I have of the Winbuilder world , you could be surprised....
You are of course perfectly free to ignore my suggestions and opinions , but doubting of my competence in the specific field doesn't help your thesis at all, and the only effect is that of unneededly irritating me.

dormient or abandoned projects - As far as possible, we will avoid deleting inactive projects.

I never proposed to delete them , ONLY, they DO NOT NEED to be backed up AGAIN, ONLY if they change (this is what differential means).


Wonko

[Nuno Brito]

Don't underestimate too much the knowledge I have of the Winbuilder world wink.gif, you could be surprised....happy22.gif
You are of course perfectly free to ignore my suggestions and opinions smile.gif, but doubting of my competence in the specific field doesn't help your thesis at all, and the only effect is that of unneededly irritating me.

You're right, my apologies as I didn't intended to irritate you.

I'm not ignoring your suggestions, but given our resources we still need to give focus on other details before moving again onto the way how winbuilder is currently structured. There are changes lined up for the winbuilder world, after 4 years there is a lot that can be improved to ease the life of developers and improve overall efficiency of used resources.

The advice, very OFTEN given on the board in reply to problems is, "start from scratch", re-download the whole project!

You're referring to a different matter. re-re-re-re-re-downloading the same file is not optimal but it is bearable to some extent as our server is well fit to support the demand and evade leech/DDoS attacks.

Here's a mug shot of our server supporting the forum and the whole load from projects and such:

In a perfect word, one would have unlimited resources, but we are not in a perfect word, and thus reducing as much as possible the needed resources or better using the ones available is only inevitable.

Yep, we're on the same brain-wave and this is an ongoing work.

[Mikorist]

We need to test the server- if someone has a strange errors ..... or 404 - 403 - or a blank page - or strange slowdown -write it here - now should be ok but we should check.....

[Gostep]

to solve the problem of file uploads, why not use a P2P system, use torrents or something. would completely remove the need for file servers.

[Nuno Brito]

to solve the problem of file uploads, why not use a P2P system, use torrents or something. would completely remove the need for file servers.

Hmm.. I actually spent a lot of time last night reading through papers of proposed peer to peer networks to see if any could be used to our help.

You see, P2P as you commonly see it on torrent sites would indeed solve the file upload problem but raise some others along way:

- Authors lose control on the number of downloads;
- Authors might lose control on the distribution of their own files;
- Users need to install/run a torrent client;
- Inside LAN's (Universities, Business, Government, etc) you'll often see torrent traffic blocked;
- Authors need to learn how upload torrents and share the links here;
- Hosting companies *hate* when their servers are used to distribute P2P;
- From our website side, we'd also need to code a forum addon to concentrate these links.

So, just by considering the above arguments you can understand why most people wouldn't see P2P as a viable option to replace attachments. P2P technology is already available for years and yet you don't see authors sharing P2P links here.

Nevertheless, I liked the idea and that was the reason why I kept on reading more and trying to see where this could be well applied for our case.

--------------

Contrary to many other forums out there, we actually have good conditions to implement a P2P algorithm custom adapted to our needs that would solve some of the nuisances described previously.

This type of feature could bring some good advantages for projects like winbuilder that require thousands of megabytes for projects managed by different teams of developers.

For example, instead of requiring people to run a p2p client, the site itself could point a direct download link to a machine hosting the file closer to the visitor that requests the download. If not the site itself, then a client would find other possible servers to use in case the main server goes offline for some odd reason.

This is the type of technology that would replace the attachment system without bringing disruption to what we already have but also adding better functionality. Similar to the P2P concept that you propose.

The current attachment system in which I'm working is not meant to be distributed across other machines but this is a nice feature suggestion to add in the future. For the moment up to December we're already filled up with work and goals to achieve but this type of things could become a nice goal for 2011.

-----------

There's a lot of interesting work out there, one of the most interesting to get ideas from was the PAST concept from Microsoft Research - http://research.micr...T/past-sosp.pdf

[Mikorist]

We need to test the server- if someone has a strange errors ..... or 404 - 403 - or a blank page - or strange slowdown -write it here - now should be ok but we should check.....

Wow - 5054 user(s) active in the past 120 minutes

and we are still keep in one piece

Test is successful

[Mikorist]

@Nuno

counter script have problem.....


i have Wating for c.statcounter.com - can not stand with this large traffic

[Nuno Brito]

Don't worry. We won't crash statcounter..