4200 crashed

wgstarks

Up for about 30 minutes and then crashed again. Console just showed the login menu and then the console option screen after logging in. I was attempting to uninstall the crowdsec pkg at the time of the crash. The uninstall appeared to be hung. 10 minutes or so with no progress removing the pkg from the shell. Maybe that’s related??? Was able to complete the uninstall process after rebooting the 4200.

Maybe the problem is heat related? All sensors are showing between 69C and 70.1C. No idea what the normal range is for this appliance.

I’m leaving a laptop hooked to the console port for a bit. Maybe it’ll catch something if it crashes again.

stephenw10

A crash report should be in /var/crash if it created one.

What happens when you try to access the crash reporter from the link in the gui?

stephenw10

@wgstarks said in 4200 crashed:

All sensors are showing between 69C and 70.1C.

That is quite hot. How is the 4200 installed? Does it have adequate airflow?

wgstarks

@stephenw10 said in 4200 crashed:

A crash report should be in /var/crash if it created one.

What happens when you try to access the crash reporter from the link in the gui?

Nothing in var/crash. Nothing ever loaded clicking the link. Just timed out.

wgstarks

@stephenw10 said in 4200 crashed:

@wgstarks said in 4200 crashed:

All sensors are showing between 69C and 70.1C.

That is quite hot. How is the 4200 installed? Does it have adequate airflow?

Ordered a 120mm 50cfm fan on Amazon. Will lay it on top as an exhaust fan.

wgstarks

This post is deleted!

Gertjan

@wgstarks said in 4200 crashed:

0 addresses deleted

Not sure what this is. Doesn't look harmful.

But this :

@wgstarks said in 4200 crashed:

gpio_open: No such file or directory
gpio_open: No such file or directory
gpio_open: No such file or directory

The start up sequence shouldn't have / use commands that ... do not exist ?
It's probably something non essential, as "gpio_open" makes me thing of the "front panel led driver".

Maybe I shouldn't compare a boot sequence of my 4100 with your 4200

Temperatures. This is without vents, and its placed above a T350 PowerEdge (ancient server running R2 2008 and all that). Not ideal, but not 70+.

stephenw10

@wgstarks said in 4200 crashed:

gpio_open: No such file or directory
gpio_open: No such file or directory
gpio_open: No such file or directory

Mmm, that does seem unexpected. Do you have anything custom installed?

Though I wouldn't expect that to be an issue.

wgstarks

@stephenw10 said in 4200 crashed:

@wgstarks said in 4200 crashed:

gpio_open: No such file or directory
gpio_open: No such file or directory
gpio_open: No such file or directory

Mmm, that does seem unexpected. Do you have anything custom installed?

Though I wouldn't expect that to be an issue.

I have a custom LED control script that lets me set color and pattern of led lights based on system status. Can’t find the link to that forum thread right now though.

stephenw10

Ah was that imported from an older device? That could certainly be throwing those gpio errors. It shouldn't cause it to crash though.

wgstarks

@stephenw10 said in 4200 crashed:

Ah was that imported from an older device? That could certainly be throwing those gpio errors. It shouldn't cause it to crash

I had a similar script on my 3100 but I don’t think it was migrated to the 4200. Either it wasn’t included in the backup file that negate tech support modified to restore from on the 4200 or it was removed by them as a precaution.

I have installed a new modified version of the led control script though. Discussion starts here. Not sure if that’s what’s causing the boot entries though. It runs on a cron schedule. Wouldn’t think that would happen until boot was complete.

stephenw10

Hmm, well it sure seems like that script still has a reference to the 2100 gpio devices that don't exist on the 4200.

wgstarks

@stephenw10 said in 4200 crashed:

Hmm, well it sure seems like that script still has a reference to the 2100 gpio devices that don't exist on the 4200.

I’ll connect via cyberduck and run a search for the old file just to be sure there isn’t still an old copy or something.

wgstarks

@stephenw10 said in 4200 crashed:

Hmm, well it sure seems like that script still has a reference to the 2100 gpio devices that don't exist on the 4200.

I'm only finding one script to control the led's and I don't think its the culprit.

#!/bin/sh
echo 0 > /dev/led/blue_0
echo 0 > /dev/led/blue_1
echo 0 > /dev/led/blue_2
echo 0 > /dev/led/red_0
echo 0 > /dev/led/red_1
echo 0 > /dev/led/red_2
echo 0 > /dev/led/green_0
echo 0 > /dev/led/green_1
echo 0 > /dev/led/green_2

check_current_states=$( pfctl -vvss | grep -e "igc3 icmp" )
WAN=0
case "$check_current_states" in
  *icmp* ) WAN=1 ;;
esac
if [ $WAN = 0 ];
then
 echo f2 > /dev/led/red_0
 echo f2 > /dev/led/red_1
 echo f2 > /dev/led/red_2
else
  /usr/local/sbin/pfSense-led.sh ready

fi

Don't see anything that looks like it matches.

stephenw10

Mmm, wouldn't expect it from that.

Does that work as expected?

wgstarks

@stephenw10
Yes. Just lights a led red when WAN status is down.

There isn’t any other custom cron entry other than an update checker so even if there was a custom script buried somewhere I don’t see how it would be called.

tgl

@wgstarks said in 4200 crashed:

Maybe the problem is heat related? All sensors are showing between 69C and 70.1C. No idea what the normal range is for this appliance.

That definitely seems high. My 4200 shows readings right around 50C, except for the "Zone 0" sensor which is typically 58C or so. It's in a closet that doesn't have much ventilation, so ambient temp maybe 25C or a bit more.

wgstarks

Looks like all my openvpn clients got deleted too. No idea what is going on with this box. Realized the problem when I tried to connect remotely. After I got home I checked and OpenVPN clients tab is empty.

This isn’t really a big problem since I’ve been thinking about switching to Tailscale anyway but I don’t know if it’s another symptom of a bigger issue?

stephenw10

Sounds like maybe the config was reverted. Or perhaps it booted an older ZFS BE if the current one became unbootable?

wgstarks

@stephenw10
Found the problem with OpenVPN after a lot of digging. Looks like dyndns wasn’t updating cloudflare even though the dyndns client tab showed that cloudflare was up to date. I could see from the dns records on the cloudflare site that they were out of date. Forced an update and now OpenVPN is reachable.

The dyndns package had the proper IP and the update script is working so perhaps this was just a temporary issue with cloudflare and only a coincidence with the timing. Not really sure. 🥺