Jump to content











Photo
- - - - -

500 Internal Server Error


  • Please log in to reply
18 replies to this topic

#1 TheHive

TheHive

    Platinum Member

  • .script developer
  • 4138 posts

Posted 06 October 2017 - 07:28 AM

Sites been down for awhile. Kept getting

500 Internal Server Error

 

Welcome Back!

 


  • Nuno Brito likes this

#2 Nuno Brito

Nuno Brito

    Platinum Member

  • Team Reboot
  • 10492 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 06 October 2017 - 09:10 AM

Have to thank Erwan. He did a wonderful work at finding what was happening wrong at the server. :thumbup:



#3 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted 06 October 2017 - 09:21 AM

Hi Gents,

 

Glad I could help :)

 

The database issue has been fixed.

 

The "500 Internal Server Error" is more difficult to trace down.

However, I am planning on spending some time on it this week end.

 

Happy to see reboot.pro back online : it is a great community.

 

Regards,

Erwan


  • Nuno Brito and netlord like this

#4 netlord

netlord

    Member

  • Members
  • 31 posts
  •  
    Germany

Posted 06 October 2017 - 09:48 AM

Happy to see reboot.pro back online : it is a great community.

 

Happy to be back ;)

I already missed you all!


  • Nuno Brito likes this

#5 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 331 posts
  •  
    Germany

Posted 07 October 2017 - 03:31 AM

and, what it is happens ?

an small report would interesting .

i am also happy to see Reboot is back :)

 

best regards

Blacky



#6 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted 08 October 2017 - 01:30 PM

and, what it is happens ?

an small report would interesting .

i am also happy to see Reboot is back :)

 

best regards

Blacky

 

1/

Initial issue was with inodes running out and log files filled with "directory index full".

This was preventing apps to create new files and filling up all log files pretty quickly.

I chased down the directory containing zillions of files, found it, and emptied the troublesome directory.

At this point the filesystem was ok and apps could again create files : no more filesystem warnings/errors.

 

2/

Second issue, because of the "directory index full" issue, the database got corrupted.

I repaired the DB.

Two (harmless) tables were permanently lost.

I had to google the definition of these 2 tables and I recreated it.

At this point, the DB was ok : no more mysql warnings/errors.

 

3/

Now, we were still encountering random 500 errors.

I believe this one has been there for a while - way before the latest mysql error.

I quickly ruled out nginx being the issue but rather focused on the PHP rendering.

As I have little confidence in the HHVM rendering engine, I reverted back to the basic PHP engine.

At this point, it looks like we are not getting any more 500 errors.

We need to look at performances thus over the coming days : HHVM is supposedly aimed at providing a better user experience (at the cost of many side issues thus...).

 

To be followed :)


  • Nuno Brito and Blackcrack like this

#7 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted 08 October 2017 - 05:10 PM

Might be me but we may have lost mail notifications ?



#8 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 331 posts
  •  
    Germany

Posted 09 October 2017 - 03:37 AM

so in short, b...sh.. goes over rabbitsh..t and the :nuke: pot was fullfilled.. :crazy:
and all must first pump it up  :rofl2:  *rofl*
 
good work, keep it and the sh.. up *bg*
:happy_dance2:
best regards
Blacky :thumbsup:
 
 
edit :

Might be me but we may have lost mail notifications ?

well, it works now again, this is important, like i think.. and the next post and mail-notification it is come certainly :)



#9 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted 11 October 2017 - 04:54 PM

About mail notifications :

After fiddling around the IPB settings I was able to send mails from the forum software (the system mail settings and PHP were fine).

 

Users should now get mail notifications like they use to before.

The mail formatting may differ from before.



#10 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted A week ago

Hi Gents,

 

The server was again running out of inodes - see this post.

I have deleted millions of empty files generated by the HHVM process.

 

I have also set a daily ask to remove these empty files (rm /var/lib/hhvm/sessions/*).

 

It should normally have no negative effect on reboot.pro (since HHVM is not used there) but if you notice strange behavior, let me know !

 

I am conscious this is a workaround but the server is hosting several web sites and I dont feel like reconfiguring these other web sites (other than reboot.pro) to have them use PHP only (and disable HHVM for good on this server).

 

Regards,

Erwan


  • misty likes this

#11 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted A week ago

For the curious techies out there who wonder how a server running out of inodes looks like.

See below.

 

When in such a state, you are not running out of space but actually out of file and directory entries : your system can no longer create any file.

 

zMf1Axz.png



#12 misty

misty

    Silver Member

  • Developer
  • 869 posts
  •  
    United Kingdom

Posted A week ago

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.

:cheers:

Misty
  • Blackcrack likes this

#13 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2249 posts
  • Location:Nantes - France
  •  
    France

Posted A week ago

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.

:cheers:

Misty

 

I will join you with a (real) glass of wine :)

 

And now time for some netflix and chill !


  • Nuno Brito and misty like this

#14 misty

misty

    Silver Member

  • Developer
  • 869 posts
  •  
    United Kingdom

Posted A week ago

I will join you with a (real) glass of wine :)


The only reason mine is a virtual glass is due to having run out of wine!

#15 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 331 posts
  •  
    Germany

Posted A week ago

huhh.. 100%use of space.. may could it intelligent, if the logs and so ..

tar cfvz /home/admin/reboot.pro/logs-$(date +%d.%m.%y-%R).tar.gz /home/admin/reboot.pro/logs/
&&
rm -rf /home/admin/reboot.pro/logs/*
&&
rm -rf /var/lib/hhvm/sessions/*

and this all week or daily in cron..
and the patient should quickly healthy in the next month ;)

and.. wth be more as 2 gig as sessions or something ? sda3 have 14Gig ..(15Gig quota i guess)
 

Thank you Erwan for your time and efforts to keep this forum running as smoothly as possible.

I'm raising a virtual glass in toast to you as I write.

:cheers:

Misty

+1 Thy erwan.l ! :))))

best regards
blacky

#16 v77

v77

    Silver Member

  • Team Reboot
  • 528 posts
  •  
    France

Posted A week ago

The only reason mine is a virtual glass is due to having run out of wine!

 
In France, we have USB Wine:

  • Nuno Brito and misty like this

#17 Nuno Brito

Nuno Brito

    Platinum Member

  • Team Reboot
  • 10492 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted A week ago

All was going well until they reached that part of "quatre-vingt-quinze" and then I got lost doing math calculations.. ;)

 

Sounds like a good USB device, anyone here already tried to boot that thing? :drunk:



#18 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13983 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted A week ago

Sounds like a good USB device, anyone here already tried to boot that thing?  :drunk:

You need a BIOS 18 years or older (21 in the US) to be legally allowed to boot booze from it.

 

:duff:

Wonko



#19 alacran

alacran

    Silver Member

  • Advanced user
  • 565 posts
  •  
    Mexico

Posted 5 days ago

All was going well until they reached that part of "quatre-vingt-quinze" and then I got lost doing math calculations.. ;)

 

What a complicated way to say 95 (better put numbers), even more complicated than in old roman numbers: VC


  • Nuno Brito likes this




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users