34 replies to this topic

[TheHive]

Sites been down for awhile. Kept getting

500 Internal Server Error

 

Welcome Back!

 

[Brito]

Have to thank Erwan. He did a wonderful work at finding what was happening wrong at the server.

[erwan.l]

Hi Gents,

 

Glad I could help

 

The database issue has been fixed.

 

The "500 Internal Server Error" is more difficult to trace down.

However, I am planning on spending some time on it this week end.

 

Happy to see reboot.pro back online : it is a great community.

 

Regards,

Erwan

[netlord]

Happy to see reboot.pro back online : it is a great community.

 

Happy to be back

I already missed you all!

[Blackcrack]

and, what it is happens ?

an small report would interesting .

i am also happy to see Reboot is back

 

best regards

Blacky

[erwan.l]

and, what it is happens ?

an small report would interesting .

i am also happy to see Reboot is back

 

best regards

Blacky

 

1/

Initial issue was with inodes running out and log files filled with "directory index full".

This was preventing apps to create new files and filling up all log files pretty quickly.

I chased down the directory containing zillions of files, found it, and emptied the troublesome directory.

At this point the filesystem was ok and apps could again create files : no more filesystem warnings/errors.

 

2/

Second issue, because of the "directory index full" issue, the database got corrupted.

I repaired the DB.

Two (harmless) tables were permanently lost.

I had to google the definition of these 2 tables and I recreated it.

At this point, the DB was ok : no more mysql warnings/errors.

 

3/

Now, we were still encountering random 500 errors.

I believe this one has been there for a while - way before the latest mysql error.

I quickly ruled out nginx being the issue but rather focused on the PHP rendering.

As I have little confidence in the HHVM rendering engine, I reverted back to the basic PHP engine.

At this point, it looks like we are not getting any more 500 errors.

We need to look at performances thus over the coming days : HHVM is supposedly aimed at providing a better user experience (at the cost of many side issues thus...).

 

To be followed

[erwan.l]

Might be me but we may have lost mail notifications ?

[Blackcrack]

so in short, b...sh.. goes over rabbitsh..t and the pot was fullfilled..
and all must first pump it up    *rofl*
 
good work, keep it and the sh.. up *bg*

best regards
Blacky
 
 
edit :

Might be me but we may have lost mail notifications ?

well, it works now again, this is important, like i think.. and the next post and mail-notification it is come certainly

[erwan.l]

About mail notifications :

After fiddling around the IPB settings I was able to send mails from the forum software (the system mail settings and PHP were fine).

 

Users should now get mail notifications like they use to before.

The mail formatting may differ from before.

[erwan.l]

Hi Gents,

 

The server was again running out of inodes - see this post.

I have deleted millions of empty files generated by the HHVM process.

 

I have also set a daily ask to remove these empty files (rm /var/lib/hhvm/sessions/*).

 

It should normally have no negative effect on reboot.pro (since HHVM is not used there) but if you notice strange behavior, let me know !

 

I am conscious this is a workaround but the server is hosting several web sites and I dont feel like reconfiguring these other web sites (other than reboot.pro) to have them use PHP only (and disable HHVM for good on this server).

 

Regards,

Erwan

[erwan.l]

For the curious techies out there who wonder how a server running out of inodes looks like.

See below.

 

When in such a state, you are not running out of space but actually out of file and directory entries : your system can no longer create any file.

 

[misty]

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.



Misty

[erwan.l]

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.



Misty

 

I will join you with a (real) glass of wine

 

And now time for some netflix and chill !

[misty]

I will join you with a (real) glass of wine

The only reason mine is a virtual glass is due to having run out of wine!

[Blackcrack]

huhh.. 100%use of space.. may could it intelligent, if the logs and so ..

tar cfvz /home/admin/reboot.pro/logs-$(date +%d.%m.%y-%R).tar.gz /home/admin/reboot.pro/logs/
&&
rm -rf /home/admin/reboot.pro/logs/*
&&
rm -rf /var/lib/hhvm/sessions/*

and this all week or daily in cron..
and the patient should quickly healthy in the next month

and.. wth be more as 2 gig as sessions or something ? sda3 have 14Gig ..(15Gig quota i guess)
 

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.



Misty

+1 Thy erwan.l ! )))

best regards
blacky

[v77]

The only reason mine is a virtual glass is due to having run out of wine!

 
In France, we have USB Wine:

[Brito]

All was going well until they reached that part of "quatre-vingt-quinze" and then I got lost doing math calculations..

 

Sounds like a good USB device, anyone here already tried to boot that thing?

[Wonko the Sane]

Sounds like a good USB device, anyone here already tried to boot that thing? 

You need a BIOS 18 years or older (21 in the US) to be legally allowed to boot booze from it.

 

Wonko

[alacran]

All was going well until they reached that part of "quatre-vingt-quinze" and then I got lost doing math calculations..

 

What a complicated way to say 95 (better put numbers), even more complicated than in old roman numbers: VC

[erwan.l]

I noticed site was getting down randomly again like previous times.

Quickly checking, volume was filled with zillions of files again by hhvm process.

Looks like the scheduled task I had set did not do the job.

Anyway, files are being removed (takes a while) and it should be ok for a good while.

 

Cheers,

Erwan

[erwan.l]

Hi Gents,

 

Reboot.pro host was feeling a bit down and needed some attention.

Took a look, did some actions and it looks ok now.

One mysql table had to be repaired : hopefully, no data has been lost.

May require some more drastic actions in the coming days/weeks.

 

I have reviewed my cron job and hopefully this time it shall do the cleanup proactively.

See post #10

 

Regards,

Erwan

[Blackcrack]

:thumpsup:

[Tokener]

Thanks a lot Erwan

[Tokener]

still no no luck uploading files to the download section.

 

An error occured with the SQL server:

mySQL query error: SELECT SUM(posts) as replies FROM ibf_topics WHERE approved=1

This is not a problem with the IPS Community Suite but rather with your SQL server. Please contact your host and copy the message shown above.

[TheHive]

Can view Topics posts again. Thanks for the repairs so far.