Jump to content











Photo
- - - - -

Intermittent failure of forum database


  • Please log in to reply
78 replies to this topic

#1 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 21 August 2019 - 09:21 PM

All this week there is intermittent failure on forum database, sometimes for some minutes other times for some hours.

alacran

Attached Thumbnails

  • Failure.png


#2 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 22 August 2019 - 11:11 AM

It was bad all week but no problems today!



#3 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 August 2019 - 12:20 PM

It was bad all week but no problems today!

till now :w00t:, I just had (the nth) 

reboot.pro Driver Error
There appears to be an error with the database.

If you are seeing this page, it means there was a problem communicating with our database. Sometimes this error is temporary and will go away when you refresh the page. Sometimes the error will need to be fixed by an administrator before the site will become accessible again.

You can try to refresh the page by clicking here

 
:(
 
:duff:
Wonko

#4 ambralivio

ambralivio

    Frequent Member

  • Advanced user
  • 195 posts
  •  
    Italy

Posted 22 August 2019 - 09:33 PM

I'm suffering the same problem, too...



#5 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 27 August 2019 - 12:07 PM

Server was in a bad state.

Gave it a reboot.

 

Some mysql tables were damaged.

Gave them a repair.

 

HHVM was taking all cpu power.

Removed old HHVM sessions.

 

Next to database hickups, mail notifications were stuck as well (fixed now).

 

Thats a quick fix but that should get it going for a while :)


  • Tokener and alacran like this

#6 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 28 August 2019 - 07:57 AM

Thanks for your efforts :).

 

But unfortunately :( it seems like the issue is not completely solved, this morning I had the same problem.

 

:duff:

Wonko



#7 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 28 August 2019 - 11:01 AM

Thanks for your efforts :).

 

But unfortunately :( it seems like the issue is not completely solved, this morning I had the same problem.

 

:duff:

Wonko

 

Indeed, spotted that.

 

There was indeed a crashed table this morning which I repaired.

I believe it was unrelated to previous issues (where i could witness that the whole server was really unstable and was in need of a reboot plus some filesystem cleanup).

 

Mysql tables crashing when used by forums like this is unfortunately a common issue.

Most of the times it is a session table left in a bad state because some http session/transaction did not complete (blame it on "the internet" and/or the forum software we are using).

To cover this I am considering to implement an automatic cron task at night to repair mysql tables.

Thus, this could be considered as bad practice and "purists" would argue that the root cause should be addressed, not the symptom ... 

I could reply then to go ahead and fix the internet :) or may be more realistically, fix the forum software.

 

In the meantime, whenever I spot this, I fix it manually.

I probably repaired table over a dozen times this last year.

Posting a message about it will also notify me but this is a catch22 situation as usually this is no longer possible to post :)

A mail directly to me then is also an option :)



#8 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 28 August 2019 - 01:30 PM

Posting a message about it will also notify me but this is a catch22 situation as usually this is no longer possible to post  :)

A mail directly to me then is also an option :)

 

Yep, but anyone with some basic politeness would probably not disturb you for such a matter, I mean it is not like you are a paid employee of a commercial company and we were paying users ...

 

... no wait, bad example :(, if that was the case we could e-mail, but the result would be an automated reply and no action taken ;).

 

:duff:

Wonko


  • Tokener likes this

#9 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 19 December 2019 - 06:29 AM

I had a few intermittent database errors yesterday.

Found a couple of tables marked as crashed...repaired them...

Seems to get better.

 

Thus, I can see some system instability (had to do a hardware reset twice) but cannot put my fingers on the root cause for now.

To be followed.


  • Tokener and alacran like this

#10 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 16 March 2020 - 09:31 PM

*
POPULAR

One mysql table was crashed which was probably preventing to view a thread and possibly to post.

Table repaired.


  • wimb, alacran and 911CD.net like this

#11 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 458 posts
  •  
    Germany

Posted 17 March 2020 - 06:11 AM

is there not a possible to make it more stable ?
with maybe a more bigger mashin/engine or something..
and hey, there gives a better raspberry.. more stronger and so..
2 ssd's (System with swap, the other /srv,
a backup posibility, external harddisk with
(the one with usb3 from WD) 5T for the next century
to backuping every week in a tar.gz on the 5T Harddisk via cron ?
(just a "putting on the table" and asking)
 
best regards
Blacky

#12 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 17 March 2020 - 08:59 AM

is there not a possible to make it more stable ?
with maybe a more bigger mashin/engine or something..
and hey, there gives a better raspberry.. more stronger and so..
2 ssd's (System with swap, the other /srv,
a backup posibility, external harddisk with
(the one with usb3 from WD) 5T for the next century
to backuping every week in a tar.gz on the 5T Harddisk via cron ?
(just a "putting on the table" and asking)
 
best regards
Blacky

 

Actually the system is quite robust (enough cpu, mem, disk, etc) and has been running for a few years now (and could probably do so for years).

 

The issue is with the software really.

By that I do not mean the O.S but the forum software i.e ip.board.

This is a (really) very old version and this is root cause for these re occuring table corruptions.

 

I am going to think of a way to automatize the database checks and repairs in the safest way possible.


  • wimb and alacran like this

#13 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 458 posts
  •  
    Germany

Posted 17 March 2020 - 09:41 AM

via cron.. guess i am.., not a bad idea, every day, all 24h .. could be a good way ..

maybe a fully script where's put all in a tar.gz for a bit logging and move it to an ftp/sftp for store ..

or so..



#14 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 17 March 2020 - 11:04 AM

This is a (really) very old version and this is root cause for these re occuring table corruptions.

No.

 

One or two versions before the one we are running there weren't these sames database errors (or they were much less frequent).

 

The issue is not with the version being old, it is with the version being bad.

 

:duff:

Wonko


  • wimb likes this

#15 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 21 March 2020 - 05:53 PM

The site has been down for a day or two.

Nothing related to a technical issue.

A file has been reported as malware and ip was locked by the hosting company...

 

File was removed, request was made to remove the "lock", and site is back.

More details about the possible reported malware here.


  • wimb and Blackcrack like this

#16 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 21 March 2020 - 06:38 PM

It is sad that a false positive results in locking of the ip by the hosting company. :(

 

It is likely that more software like this can give similar trouble in the future ..... :unsure:

 

Anyway for most of us the software giving the trouble is outdated so that the removal is not a problem for me.  :)

 

Thanks for the removal and making reboot.pro online gain  :)

 

:cheers:

 

EDIT: I have removed VHD_XP_Setup and Win_Restore, since they might cause similar false positive problem  :)



#17 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 21 March 2020 - 08:46 PM

@ wimb

 

Remember some time ago I suggested to apply a password to your uploads to avoid this kind of troubles, you were doing it for some time and didn't have more false possitives, but later you stopped doing it.

 

alacran 



#18 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 21 March 2020 - 09:32 PM

Remember some time ago I suggested to apply a password to your uploads to avoid this kind of troubles, you were doing it for some time and didn't have more false possitives, but later you stopped doing it.

 

 

In VHD_W7_Compact and VHD_W8_Compact  I am still using the password protection method that you proposed, which by the way is working good.

 

The problem of false positive is NOT present in the new projects, so there is no need for password protection and that is preferrable.

 

The problem of false positives did exist in the very old XP related software, but that software has now completely been removed from reboot.pro.

 

So hopefully there will be no problems anymore with my software here at reboot.pro  :)

 

False positives also occurs nowadays for WinNTSetup4 ..... and there is may be more software around here at reboot.pro which will give false positive problem.


  • alacran likes this

#19 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 March 2020 - 08:53 AM

If anyone has a twitter or github account, it may be a good thing to contact the "amateur botnet hunter" that managed to send the advise and single handedly disrupt the whole board (to thank him for his contribution in making the Internet so much safer):

https://twitter.com/zbetcheckin

https://github.com/zbetcheckin

 

I am still intrigued on how exactly the whole stuff works.

 

It seems like - more or less - it works like this:
1) someone (possibly through a script) downloads random files from random sites
2) it uploads them to VirusTotal (this file has a rating of 20/73, so it is not particularly "bad" or however has high probabilities of being a false positive)

3) then a report is sent to abuse.ch

4) the host of reboot.pro takes down the WHOLE site (maybe with a single day advance notice?)

 

Is it actually working like this? :w00t: :ph34r:

 

:duff:

Wonko



#20 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 22 March 2020 - 12:51 PM

If anyone has a twitter or github account, it may be a good thing to contact the "amateur botnet hunter" that managed to send the advise and single handedly disrupt the whole board (to thank him for his contribution in making the Internet so much safer):

https://twitter.com/zbetcheckin

https://github.com/zbetcheckin

 

I am still intrigued on how exactly the whole stuff works.

 

It seems like - more or less - it works like this:
1) someone (possibly through a script) downloads random files from random sites
2) it uploads them to VirusTotal (this file has a rating of 20/73, so it is not particularly "bad" or however has high probabilities of being a false positive)

3) then a report is sent to abuse.ch

4) the host of reboot.pro takes down the WHOLE site (maybe with a single day advance notice?)

 

Is it actually working like this? :w00t: :ph34r:

 

:duff:

Wonko

 

Looks like it works like that indeed.

 

To me, looks like a false positive on top of a strong reaction from the hosting company.

Thus, I believe a warning was sent by the hoster.

 

Took me a while to understand what was going on : no more http, no more ssh, no improvement after a reboot triggered via the hoster web panel to finally find out (after going thru all menus in the web panel) that the machine had been "locked" indicating which file on reboot.pro had to be removed before I could request the machine to be "unlocked".

 

Lesson learned :)

Hopefully this is a one off.

 

Putting a password on the zip file is a good alternative.

Checking the files against virustotal before uploading is also recommened even if what is whitelisted today could be blacklisted tomorrow...

And ideally the forum software should have some sort of AV feature but our old software does not have such option.

 

Being locked myself at home (tough times for us all these days and a special attention to my itilian and spanish fellows who entering a peak situation as it seems...), this keeps me busy and makes me forget this sad situation.



#21 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 March 2020 - 01:26 PM

Checking the files against virustotal before uploading is also recommened even if what is whitelisted today could be blacklisted tomorrow...

And ideally the forum software should have some sort of AV feature but our old software does not have such option.

Historically AutoIt files have been marked (falsely) positive on and off by different antivirus, the point remains that IF 20/73 (or 27.94%) is considered "consensus" having a single antivirus connected to the board (which has obviously more probabilities of being 1 out of the 53 remaining) is completely m00t.

 

Though I find it a very useful site for informative purposes only, Virus Total cannot be considered as reliable, particularly it lists tens of antivirus which are total crap.

 

It is not the specific case at hand, as the specific file has been tagged by more than one "known/reliable" tool[1], but let's say that it was the same 20/73 flagged by the unknown/crappy ones ...

 

:duff:

Wonko

 

[1] even if there is not any recognized "badge" for that 



#22 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 22 March 2020 - 01:49 PM

Though I find it a very useful site for informative purposes only, Virus Total cannot be considered as reliable, particularly it lists tens of antivirus which are total crap.

 

It is not the specific case at hand, as the specific file has been tagged by more than one "known/reliable" tool[1], but let's say that it was the same 20/73 flagged by the unknown/crappy ones ...

 

 

This is one weak point of VT.

Very useful tool, nice and easy API to use (I guess I could even interface it with our forum software) but no ability AFAIK to choose which "players" will be used or not.

Clearly and I aggree, some of the AV's listed on VT are useless.



#23 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 22 March 2020 - 02:50 PM

It seems actually almost all AVs consider x86 software created with Autoit as a virus, I think sooner or latter this AVs behaviour is going to include x64 software made by Autoit.

 

So then as a long term solution and to be on the safe side, I suggest all uploads including any Autoit software (even only x64) should be password protected.

 

Why wait to have troubles if we already know the fix?

 

alacran


  • wimb likes this

#24 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 22 March 2020 - 05:51 PM

It seems actually almost all AVs consider x86 software created with Autoit as a virus, I think sooner or latter this AVs behaviour is going to include x64 software made by Autoit.

 

So then as a long term solution and to be on the safe side, I suggest all uploads including any Autoit software (even only x64) should be password protected.

 

Why wait to have troubles if we already know the fix?

 

 

Sure I will do that  :)



#25 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 23 March 2020 - 06:57 AM

All downloads of my software at reboot.pro are now Encrypted and protected with password = bootwimb

 

:cheers:






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users