Random reboots since upgrade to 2.9.0

musicwizard

Since i updated to CE 2.9.0 a few days ago my router did reset out of nowhere random 3 times already. Just now a few minutes ago it did it again. Before i never had this update. I never changed anything in the config only updated it.
I looked in my logs but could not find anything that could result in this random reboot. Its 2 time in about 30 hours timespan.
i was running stable on 2.8.1 since that first upgrade more then 230 days i think without a single problem.
Now with this 2.9 it already rebooted a few times randomly.

musicwizard

it is possible that it has something to do with AMD-CPPC which is changed from 2.8.1 - 2.9.0
now the setting machdep.hwpstate_amd_cppc_enable="1"
i still need to test and see when i put it on 0 will fix it
As i have a AMD ryzen 5600U as router.

stephenw10

No crash report? Anything at the console when it rebooted?

musicwizard

@stephenw10
No logs no crash rapport. could not find anything

stephenw10

Hmm, tricky to diagnose. Usually a hardware issue but, yes, it could be some new hardware feature that's used by a driver update.

luckman212

I've been having a "crash" of sorts too where both my ixl interfaces (WAN/LAN) die, which fairly quickly causes the entire unit to become inoperable (console ports get hung up trying to process the menu I think…). Not Netgate hardware, this is a Protectli VP6670.

I've been fiddling with lots of knobs over the last week trying to stabilize that (it's happened 3 times now). hwpstate_epp was definitely one of them. Mine is an Intel CPU not AMD, but I had adjusted Speed Shift to 20. I've since toned it back down to the default of 80. Also having a fiddle with disabling PCI ASPM on those NICs, and making sure TSO/LRO are disabled.

Hard to say if it's a new 26.07/2.9.0 thing since this box has only ever run 26.07, and before that my trusty 6100 ran for years without a hiccup.

stephenw10

It just became non-responsive though? No reboot by itself?

luckman212

@stephenw10 right, it didn't reboot- I hooked up the HDMI cable and saw the console was still alive, but interfaces were dead and even the serial console locked up - I think a hanging command caused that. One reason I thought it was related to hwpstate was, I tried running sysctl -a and it hung right here:

the next line would have been...

dev.hwpstate_intel.9.epp

which corresponded to 1 of the cpu cores that was hanging.

stephenw10

Hmm, that does seem likely....

musicwizard

@stephenw10 That is always possible but 2.8.1 was running since the update and had uptime of 230 days. and i dont have this hardware that long yet. i'll let it run and see what happens.

musicwizard

@stephenw10 i did change what i said above but didnt not restart unless it shortly after i did that it did 1 i didnt notice.
But anyway its been running for 2 days now which is the longest on 2.9.0 So we'll see. i hope this solved it as i didnt change anything else in the setup.

luckman212

One other thing I recently changed (fingers crossed that it was actually the root cause and is now solved… have not had the issue in a few days) is disabling PCIe ASPM on the ixl NICs. I adapted someone else's script for this, sharing it below in case anyone finds it useful:

https://gist.github.com/luckman212/b1d0cbfdb9b018fcb679c5bfb3609e3a