www.zeroshell.org Forum Index www.zeroshell.org
Linux Distribution for server and embedded devices
 
 SearchSearch  RegisterRegister  UsergroupsUsergroups 
 ProfileProfile  Log inLog in  Log in to check your private messagesPrivate Message 

Frequent ZeroShell crashes, version 3.4.0

 
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> Signal a BUG
View previous topic :: View next topic  
Author Message
TheNanny



Joined: 06 Mar 2013
Posts: 17

PostPosted: Mon Feb 01, 2016 10:24 am    Post subject: Frequent ZeroShell crashes, version 3.4.0 Reply with quote

Hello all.

Last weekend I tried to update my ZeroShell-router from Version 3.2.1 to 3.4.0.
So via the webinterface I shut ZeroShell down (after an uptime of more than 220 days) and replaced LiveCD 3.2.1 with LiveCD 3.4.0.
It's runnning on an Intel PC.

Now I have the problem, that the router reboots very often. Once it was running for 2 hours, but sometimes just for 10 minutes.
I think the reboots are caused by system crashes, so I checked all logs. I can only find log entries for the boot sequences of ZeroShell, but nothing about what happens before to cause the restarts.

Then I tried version 3.3.2, same behavior.

After going back to version 3.2.1 the Router runs stable again, no reboots, the uptime right now is about 15 hours.

Can someone help me with finding a solution?

Thank you in advance.
Back to top
View user's profile Send private message
TheNanny



Joined: 06 Mar 2013
Posts: 17

PostPosted: Tue Feb 09, 2016 6:53 am    Post subject: Reply with quote

Hello.

Unfortunately, I still have no solution.
Can anyone help me?
Back to top
View user's profile Send private message
TheNanny



Joined: 06 Mar 2013
Posts: 17

PostPosted: Tue May 10, 2016 10:26 am    Post subject: Reply with quote

Hello.

I now tried the latest version 3.5.0, but still the same behavior. ZS reboots every 2-16 hours.
Since a few days I again try to debug the crashes with the help of log files and all I can find on the issue on google. It looks that in the moment of the crash the kernel can't do anything more but reboot. No entries in log files, no crash dumps, no messages or any data on SSH connection (for example dmesg outputs).
SSH connections are not closed by ZS in the moment of crash. When I try to use the SSH terminal after the crash it seems to be still connected, and it comes with the message "pipe broken".
I have some experience with linux systems. I never encountered a behavior like this before. Crashes of Linux are very rare and I never encountered a crash without any hints in log files or on the screen just like pushing the reset button. If I had to guess I would assume a hardware failure of the PC. But when I boot the system again with the 3.2.1 version, everythings is perfect again.

For my needs, ZS is best router distribution available (thank you Fulvio for your work). I also want to benefit from secrity issue fixes so I would be really glad if someone could help me with this.
Back to top
View user's profile Send private message
squigley



Joined: 26 May 2016
Posts: 14

PostPosted: Thu May 26, 2016 7:59 pm    Post subject: Reply with quote

This was happening to me. I had a VGA monitor connected so I would see the kernel crash output. From that I deduced it was being caused by the NIC I had in there, a Compaq ThunderLAN dual port card.

Once I took that NIC out and replaced it with a single port 3Com NIC, it stopped crashing and rebooting.
Back to top
View user's profile Send private message
alaust



Joined: 07 Jun 2016
Posts: 1

PostPosted: Tue Jun 07, 2016 7:11 pm    Post subject: Reply with quote

i have the same issue
Back to top
View user's profile Send private message
TheNanny



Joined: 06 Mar 2013
Posts: 17

PostPosted: Thu Jun 16, 2016 6:41 am    Post subject: Reply with quote

Hello all.

Since about a month I'm running the ZeroShell 3.5.0 now without crashes (uptime is now about 28 days). I'm still not sure why ZS 3.2.1 was stable on my PC and later versions crashed frequently.
My problem while debugging the crashes was that they occurred randomly. It was no option to sit in front of a monitor waiting for a possible crash related output because there were no crashes when I waited for them in front of the monitor...

Since I assumed from the beginning that the reboots were caused by crashes (as you know in Linux kernel they are called 'kernel panics') I read a lot about debugging those kernel panics.
I still found no way to catch the panic output and I also found no way to activate a crash kernel like on an Ubuntu system, which handles the panic outputs of the kernel and writes them to the log files.
So I had another idea. I installed ZS to HDD and switched the console output from VGA to serial interface. Then I connected the serial port of my router with a null modem cable to a second PC running a terminal session. So I was able to write all of ZS console outputs to a file and finally I could catch the kernel panic output.

The kernel panics were caused by MCEs (machine check exception). I googled a lot and learned that MCEs are usually caused by hardware malfunctions. So I ran a lot of tests on the routers' hardware without any errors.
Then (after I googled even more) I found the hint, the MCEs also can occur because of bugs in the Intel processors firmware called 'microcode'. One way to update this Intel microcode is upgrading the BIOS and I really found a newer BIOS version for the routers' main board containing a newer version of the microcode.
After updating the BIOS the router runs stable with ZS 3.5.0 and the frequent crashes are gone.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> Signal a BUG All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group