First hard crash in years on pfSense
-
There's a new BlinkBoot version that was just released in a new Netgate Firmware Update package
CORDOBA-03.00.00.03t. I don't believe it will do anything here but it would be an easy test. The updated package is only in 23.01 though. -
@stephenw10 Noted
-
@stephenw10 Well, my six weeks test period has now concluded and the box has been completely stable on 22.05 during that period.
So tommorow I’ll give a full power-off + disconnect of my USB serial Port device a spin, and let it boot on 23.01 again.
Here’s crossing my fingers that this will cut it. Otherwise I’ll have to start testing without my SSD and my SFP optics.
I get that FreeBSD 14 may theoretically use some region of memory or cache or a new instruction that 12.3 does not, and thus hit an actual hardware error that 12.3 just never triggers. But I find that pretty unlikely…..
-
Yeah, it does seem unlikely. Mostly because there are thousands of 6100s running 23.01 and not hitting it.
-
@stephenw10 said in First hard crash in years on pfSense:
Yeah, it does seem unlikely. Mostly because there are thousands of 6100s running 23.01 and not hitting it.
Well, 10 days in and it crashed again on 23.01…
When i booted it 10 days ago I made sure it had a full power off, and my serialport USB device is not plugged in.
It’s still a MCA error - but really really strange it always takes about 8-12 days for it to crash - and that it is 100% stable in 22.05.What’s the best course of action now? Test with no SFPs or remove my SSD and install/boot from eMMC?
Or should I wait and try a full repave with 23.05 once released? -
Hmm, it would be good to test 23.05 but it would not surprise me at all if the issue exists there too.
It must be some hardware difference so, yes, if you can I would test without the SFP modules.
-
@stephenw10 Just curious, But does 23.05 contain a newer FreeBSD 14 kernel and driver/module versions (fixes) than 23.01?
Or is 23.05 only the fixes netgate has made to various services and components? -
It has newer drivers. It's built on a newer FreeBSD head snap.
-
@stephenw10 Okay - I’ll make sure to give 23.05 a spin first to see if that changes anything.
-
@stephenw10 At the risk of jinxing the situation it seems 23.05 makes a difference. It has been completely stable for 23 days now - no issues since the upgrade, and before it would never go more than about 14 days without a crash.
I continue to be 100% certain that even though the crashes in 23.01 reported defective hardware, it is not the case.
22.05 never crashed on me and it started the day i upgraded to 23.01 - returning to 22.05 made it stable again.
So far 23.05 seems to make it stable again. (Fingers crossed) -
Hmm, bizarre.
Really the only way I can see that happening is somthing in 23.01 that's tickling some hardware device that's marginal on your particular 6100. And than is now removed from 23.05.
That's a lot of things that have to line up... so perhaps I'm overlooking something.
-
@stephenw10 said in First hard crash in years on pfSense:
Hmm, bizarre.
Really the only way I can see that happening is somthing in 23.01 that's tickling some hardware device that's marginal on your particular 6100. And than is now removed from 23.05.
That's a lot of things that have to line up... so perhaps I'm overlooking something.I agree, but to be fair I may be an outlier as I have installed a middle tier 512Gb Trandcend SSD myself to prevent death by write exaustion.
Also using two 1Gbe SFPs in ix2 and ix3. So not your average 6100 install. -
@keyser said in First hard crash in years on pfSense:
... I have installed a middle tier 512Gb Trandcend SSD myself to prevent death by write exaustion.
Also using two 1Gbe SFPs in ix2 and ix3. So not your average 6100 install.It does not seem that extreme. The SSD size is above the manufacturer's suggested 256GB limit but I don't see that changing anything of note, especially as they run slower due to the interface limits.
Running SFPs in the ix interfaces will probably lower the heat & power usage compared to SFP+ use too.
The device itself is designed to run with things like wifi and LTE 4G/5G radios installed, so as shipped by Netgate it has board-capacity to spare.
I run SFP+ DACs at 10 GbE and up to two 2.5 GbE ports, plus a pure Optane 64GB NVMe in my 6100.
️ -
@RobbieTT Yeah, I agree. But my particular brand of SFPs or the SSD model and make might be the cause here - And I’m likely the only user in the world with this hardware combination - or in fact user of that SSD model in a 6100….
-
Indeed, it really doesn't seem that extreme at all.
-
@stephenw10 At the risk of jinx'ing it, I'm going to declare the situation resolved by upgrading to 23.05. My box has been 100% stable for 34 days now which is 20-30 days more than it ever would on 23.01 before crashing with a hardware error.
So something is better in 23.05 on my particular 6100 :-)Looking forward to 23.05.1 with the IPsec fix it includes :-)
