After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead

SteveITS

@MordyT I would try:

Check if the console (cable) shows any errors at the time
check storage write life: https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html
try a reinstall: https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

MordyT

@SteveITS said in After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead:

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html

All good on that front, apparently under 10% used

@SteveITS said in After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead:

Check if the console (cable) shows any errors at the time

That's going to be tricky, but I'll work on it. Any other way to get the output?

@SteveITS said in After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead:

try a reinstall: https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

I've opened a ticket for the image.

MordyT

Reflashing the image and restoring the backup seems to have stabilized the system.

Of note, you can't use just any flash drive - a 5 year old 64GB Kingston USB3.0 drive would crash during the run recovery stage and I had to dig out a 8GB USB2.0 drive for it to work.

I still don't have boot environments though, I thought a reflash brought it to ZFS?

bigsy

@MordyT ZFS is not supported on ARM32 appliances (Netgate 3100 and 1000).

SteveITS

@MordyT said in After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead:

Reflashing the image and restoring the backup seems to have stabilized the system.

The brute force method, but it's in a known good state going forward.

Re: console, I was hoping it would show something at the time of the crash, maybe a disk error or something. Doesn't sound disk related though so that's good.

rcoleman-netgate

@bigsy said in After upgrade to 23.05.1, pfSense on Netgate 3100 basically dead:

@MordyT ZFS is not supported on ARM32 appliances (Netgate 3100 and 1000).

This is 100% accurate.

MordyT

Unfortunately, about 30 hours post flash, the issue has returned.

All traffic to the WAN (dedicated WAN port on the box) just dies. SSHing to the box shows that WAN1 no longer has an IP or anything assigned. I can set my gateway to WAN2 and have connectivity - WAN2 is in OPT1.
The pfSense box responds to ping, SSH, but the WebUI is either dead or very slow to respond. If I reset option 16 on the console/SSH, I can get it back for a few minutes, then it starts to 504 on me. Option 11 does not help at all.

What logs should I looking for? How should I troubleshoot this?

There are no errors displayed on the console.

SteveITS

@MordyT is something stuck/using CPU if you run top? Your note and the PHP error above sound like PHP is not responding.

MordyT

@SteveITS
Looks ok to me...

I have a ton of this in dmesg though

That is the WAN1 interface.

SteveITS

@MordyT mvneta2 is the WAN port so that matches what you're saying originally. Sounds like you tried the same patch cable on a laptop. I'd try replacing it anyway just to see. Another thing that helps some people is to put a switch between pfSense and the ISP router so the link stays up as far as pfSense knows.

stephenw10

What's the WAN actually connected to? Is it actually losing link like that?
Not that the WAN flapping should cause php to get hung up like you're seeing.

What's in the main system logs when that happens?