Upgrade to 21.02-RELEASE borked on SG-3100

DeltaOne

@lnguyen said in Upgrade to 21.02-RELEASE borked on SG-3100:

@deltaone Just issue the command above and reboot. Once an official patch is released, you can issue the rm command and reboot.

Have done so. Thanks.

rsherwood_va

@rsherwood_va I have been running smoothly since the original problem. I just had a lockup this morning - tried creating /boot/loader.conf.local and adding hw.ncpu=1, then rebooting as recommended by the team here, but that did not resolve the issue. As before, the following process worked:

ssh in to the router (you may need to use the IP address)
choose option 11 (Restart webConfigurator)
log in to the web gui
(optional - not sure if this helped) Under status -> services, start the DNS and DHCPD services
Under system-> setup wizard. Accept all the answers (it will remember what you chose last time, except for the admin password - reenter the old one)

rerunning the PFSense first-time setup wizard (accepting all the previously chosen values and choosing to enter the existing admin password as the "new" admin password) resolved the issue - I am now running smoothly.

I haven't seen the new update yet, so I'm assuming the issue is trickier than they thought. If you are stuck, try running the wizard again.

yaminb

Another update.

With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
It feels like ncpu=1 has helped, but I don't think it's the issue

I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

bmeeks

@yaminb said in Upgrade to 21.02-RELEASE borked on SG-3100:

Another update.

With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
It feels like ncpu=1 has helped, but I don't think it's the issue

I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

The switch to using a single CPU is a workaround that minimizes the chance of hitting the bug, but it does not eliminate the chance.

The actual problem has been identified and a fix is being tested. Here is a link to the discussion by the FreeBSD kernel programming nerds* of the problem and the fix: https://reviews.freebsd.org/D28821. I believe the pfSense team is now vigorously testing images with this fix applied to be sure the fix is really "the fix". And from the activity on the Redmine bug site for pfSense, it looks like a few other bugs are being addressed as well.

Note -- I don't mean "nerds" in an insulting sense . But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

RobertBrooks

How likely is this issue specific to the SG-3100?

My faithful SG-2400 has required a few reboots since the upgrade,
after never needing to be restarted in its five years of service.

                         -- Rob

bmeeks

@robertbrooks said in Upgrade to 21.02-RELEASE borked on SG-3100:

How likely is this issue specific to the SG-3100?

My faithful SG-2400 has required a few reboots since the upgrade,
after never needing to be restarted in its five years of service.
                         -- Rob

An SG-2400? Do you maybe mean the SG-2100? If 2100, that is also an ARM CPU, but it is a 64-bit version.

RobertBrooks

@bmeeks

I misspoke; I meant an SG-2440, which I recognize has an Atom CPU; I didn't know if this problem is specific to the
ARM.

                                        -- Rob

lohphat

@bmeeks said in Upgrade to 21.02-RELEASE borked on SG-3100:

Note -- I don't mean "nerds" in an insulting sense . But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

I've worked with kernel and compiler hackers and they are indeed a breed apart. You have to be able to abstract activity across interrupts, CPU rings, and microcode idiosyncrasies. It's the difference between playing checkers and 4D chess.

lohphat

Note there is now a 21.02_1 release for SG-3100 to fix the FreeBSD kernel bug causing the instability.

https://forum.netgate.com/topic/161421/pfsense-plus-21-02-p1-now-available/11

DeltaOne

@lohphat

We applied the 21.02_1 update about 12. hours ago. So far...all is good, our 3100 is running normally and back on two CPUs.

Many thanks to the Netgate teams for this quick fix!

JGdgZPQatDDjpA

@jgdgzpqatddjpa

Just upgraded my 3100. No issues so far.

shadtheman

Upgraded to 21.02.1 yesterday, ran 14 hours then hard lockup. Anyone else seen this?

lnguyen

@shadtheman Yes. One of my remote locations had a lockup after 12+ hours today around 1:15PM PST. Unfortunately no smart hands to get any useful information from console.

sabennett

@deltaone Same here. No issues, running smooth. I’ve also added a 2 WG clients.
I plan on adding pfBlocker-ng tomorrow.
And let that settle in for a couple days.

The Netgate support staff have been nothing less then awesome to me, and for that I Thank-you.

mcury

@sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

I plan on adding pfBlocker-ng tomorrow.

Don't install pfBlockerNG just yet.

https://redmine.pfsense.org/issues/5413
https://redmine.pfsense.org/issues/11444

nonvtec

@mcury Thanks for all the intel, I'm holding out for a while longer.

lohphat

@mcury said in Upgrade to 21.02-RELEASE borked on SG-3100:

@sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

I plan on adding pfBlocker-ng tomorrow.

Don't install pfBlockerNG just yet.

https://redmine.pfsense.org/issues/5413
https://redmine.pfsense.org/issues/11444

11444 is fixed.
5413 has to do with the unbound DNS resolver having issues and has been an open bug for 5+ years.

The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

mcury

@lohphat

They will fix 5413 first...

sabennett

@mcury Ok thanks for the heads up and the links.

mcury

@lohphat said in Upgrade to 21.02-RELEASE borked on SG-3100:

The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

I opened a TAC for the pfblockerNG problem and that was their answer..

"The package dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:
https://redmine.pfsense.org/issues/5413"

Ticket was marked as resolved yesterday..