Upgrade to 21.02-RELEASE borked on SG-3100

RobertBrooks

I misspoke; I meant an SG-2440, which I recognize has an Atom CPU; I didn't know if this problem is specific to the
ARM.

                                        -- Rob

lohphat

@bmeeks said in Upgrade to 21.02-RELEASE borked on SG-3100:

Note -- I don't mean "nerds" in an insulting sense . But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

I've worked with kernel and compiler hackers and they are indeed a breed apart. You have to be able to abstract activity across interrupts, CPU rings, and microcode idiosyncrasies. It's the difference between playing checkers and 4D chess.

lohphat

Note there is now a 21.02_1 release for SG-3100 to fix the FreeBSD kernel bug causing the instability.

https://forum.netgate.com/topic/161421/pfsense-plus-21-02-p1-now-available/11

DeltaOne

@lohphat

We applied the 21.02_1 update about 12. hours ago. So far...all is good, our 3100 is running normally and back on two CPUs.

Many thanks to the Netgate teams for this quick fix!

JGdgZPQatDDjpA

@jgdgzpqatddjpa

Just upgraded my 3100. No issues so far.

shadtheman

Upgraded to 21.02.1 yesterday, ran 14 hours then hard lockup. Anyone else seen this?

lnguyen

@shadtheman Yes. One of my remote locations had a lockup after 12+ hours today around 1:15PM PST. Unfortunately no smart hands to get any useful information from console.

sabennett

@deltaone Same here. No issues, running smooth. I’ve also added a 2 WG clients.
I plan on adding pfBlocker-ng tomorrow.
And let that settle in for a couple days.

The Netgate support staff have been nothing less then awesome to me, and for that I Thank-you.

mcury

@sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

I plan on adding pfBlocker-ng tomorrow.

Don't install pfBlockerNG just yet.

https://redmine.pfsense.org/issues/5413
https://redmine.pfsense.org/issues/11444

nonvtec

@mcury Thanks for all the intel, I'm holding out for a while longer.

lohphat

@mcury said in Upgrade to 21.02-RELEASE borked on SG-3100:

@sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

I plan on adding pfBlocker-ng tomorrow.

Don't install pfBlockerNG just yet.

https://redmine.pfsense.org/issues/5413
https://redmine.pfsense.org/issues/11444

11444 is fixed.
5413 has to do with the unbound DNS resolver having issues and has been an open bug for 5+ years.

The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

mcury

@lohphat

They will fix 5413 first...

sabennett

@mcury Ok thanks for the heads up and the links.

mcury

@lohphat said in Upgrade to 21.02-RELEASE borked on SG-3100:

The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

I opened a TAC for the pfblockerNG problem and that was their answer..

"The package dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:
https://redmine.pfsense.org/issues/5413"

Ticket was marked as resolved yesterday..

lohphat

@mcury Your ticket may be resolved as is usual because the cause of your support ticket is known, but the defect itself -- tracked separately -- is still open. There's no indication they're close to fixing 5413 yet and I haven't seen the new bug tracking for the php crashes yet over on the FreeBSD tracking site.

Each "scope" have their own tracking systems. 1) NetGate Support 2) pfSense bug tracking and 3) FreeBSD bug tracking.

They are handled by different independent orgs -- it was pretty impressive for the FreeBSD 11444 bug to be addressed so quickly but it was due to its criticality.

pfSense 5413 has been open 5 years and the status only shows a "target" for fixing but that's not a commitment of timing.

So my expectation is there will be a short term work around instead of a permanent fix in the near-term.

mcury

@lohphat Yes, ticket marked as resolved yesterday was to indicate that it's a recent feedback from Netgate and they are working on it.
But thanks, personally I didn't know the bug tracking flow and how it works

They are working on it, I'll be checking the redmine for news during this period

lohphat

[cross-posting is usually frowned upon, but I wasn't sure if I could link the idential posts so I'm manually copying it here]

FYI there are two new redmine bugs to track the behavior being seen. Both are related to the FreeBSD php bug.

https://redmine.pfsense.org/issues/11466 "Snort exit with sig 11 on SG-3100"

https://redmine.pfsense.org/issues/11551 "SG-3100 with pfBlockerNG doesn't pass traffic"

This MAY be the tracking bug for the php crash at it was a recent report with FreeBSD 12.1 but the new pfSense 21.02 is using FreeBSD 12.2. The last comment asks if it indeed is a continuing issue on 12.2:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244049

bldnightowl

This post is deleted!

morgenstern

Hey guys, I did an upgrade from 2.4.5 build to 22.01 on my spare SG-3100 unit and now I am experiencing a mixture of the above mentioned issues. But mainly I cannot access it via LAN and the unit itself doesn't have internet although it can ping the upstream gateway. I tried downloading the 2.6 community pfsense software and do a memstick reinstall but it failed saying it cannot read some binary file. Is there an official way to get a copy of the pfsense+ software without the support subscription in place?

SteveITS

@morgenstern said in Upgrade to 21.02-RELEASE borked on SG-3100:

official way to get a copy of the pfsense+ software

Open a free ticket at www.netgate.com/tac-support-request and they will give you a link to the 3100 firmware and instructions.

pfSense CE can't install on the ARM CPU.