21.02 Sudden lockup

rloeb

@lnguyen Nice catch. The command did not create the file. My Linux is pretty feeble these days, so I'm unclear what to do next. I'll see if I can pull one of my techies off what they're doing and he can chase this. I'll go back to running the company, which is all I'm competent to do.

kphillips

@lnguyen You are correct. I've updated my original post.

kphillips

@rloeb Snort is broken on the SG-3100 and pfSense Plus 21.02. This is due to a bug in the package, not pfSense Plus. Snort has some badly coded components that Intel CPUs usually just "auto fix", but on ARM that mechanism doesn't exist. As such, something appears to have broken during the move to 21.02. We have a bug report for that, but if you need snort you'll want to be on 2.4.5p1 until that is sorted. Otherwise, Suricata works fine on the SG-3100 AFAIK right now on 21.02.

rloeb

@kphillips Thank you. Good to know. Going to try to tough it out. Alternative is to switch to a gateway router with no filtering, just to keep folks productive.

styxl

@kphillips did they block all packages from 2.4.5-p1? i rolled back and restored from 21.02 now i cannot re-install all the packages for 2.4.5-p1 what is going on?

SteveITS

@styxl Did you set System/Update/Update Settings to "previous stable version (2.4.5)"? I can see packages on a 3100 that wasn't upgraded.
You might try https://docs.netgate.com/pfsense/en/latest/troubleshooting/pkg-broken-database.html.

kphillips

@styxl Make sure you select "Previous Stable Version (2.4.x)" under System --> Update. The repos for 21.02 are now offline to keep people from upgrading to it for the SG-3100 right now.

styxl

@teamits i just did and it worked, thx

alpharulez

Does anyone know if Suricata on 21.02 is impacted the same as Snort? Thanks!

stephenw10

On the SG-3100 it would be, in blocking mode at least. Like Snort it has to reload the ruleset whenever a new IP is added to the block table.

Steve

alpharulez

@stephenw10 ok thanks for the response
Will hold fire.

stephenw10

....unless you're seeing this: https://redmine.pfsense.org/issues/11466
That applies to Snort only.

router

Hello, is there an update on this issue?

I'm experiencing major packet loss and unable to download new packages.

I've already added hw.ncpu=1 to /boot/loader.conf.local.
This had no noticeable affect.

Our systems are completely degraded by this issue. We cannot handle the risk and downtime required to reinstall. This is a major impact for us.

Thanks...

kphillips

@router Packet loss is not a symptom of this issue. The SG-3100 would completely freeze up and force a reboot. If you have packet loss, its not 21.02 most likely. Check your gateway monitoring.

bldnightowl

Well, add me to the list of people that downgraded back to 2.4.5p1. And that was quite a hassle/nightmare by itself. Even with pfBlockerNG removed (which was an unsustainable solution for any period of time, of course), the system was still freezing or behaving erratically at times. Getting the packages back to the way they were pre-21.02 did not automatically happen as it should have -- and I had to manually intervene several times. This failed upgrade cost me a couple of days at least of my time, and like others I am very unhappy about that.

I am a software engineer too and understand how very hard it is to test field configurations for an extremely customizable product. So I'm not trying to make anyone at Netgate fill badly --- but this was pretty disastrous for many users, and a a detailed post mortem explaining what went wrong, why and how it will be avoided in the future would be hugely appreciated. For example, it appears your QA did not have pfBlockerNG(-devel) (which I would be willing to guess is in very widespread use) properly in its standard performance testsuite. I hope that's been rectified.

Thanks for the hard work and responsiveness when things did blow up, particularly you moderators on the front lines absorbing all the screams from your users. And especially to those of you responding while impacted by the much worse disasters in Texas.

kphillips

In case anyone is wondering what the root cause of the SG-3100 locking up was, here is the FreeBSD compiler issue that has been fixed and will be used for the fixed release when it comes out. Dev team has been working hard over the weekend on this one.

https://reviews.freebsd.org/D28821

nick108

@kphillips I have to add my name into the packet loss issue. I've had this SG-3100 since approx 2019 and across multiple ISPs I've only had one instance of packet loss and that was not pfSense related. After disabling pfBlockerNG-devel I have so far had 1 or 2 complete lockup and today had 2 instances of 90%+ packetloss over my IPV4 main gateway and the overlying IPv6 over v4 tunnel which exits the same gateway. No CPU spikes that I could see.

kphillips

@bldnightowl Even if we had tested pfBlockerNG-devel, it wouldn't have caused the issue unless the firewall was under moderate to heavy load. This was never pfBlockerNG's fault, but was a problem with the filter reload which pfBlockerNG was triggering more often than was normal. I expect we'll be adding more packet-driven stress tests to our list of things to do in any future releases and will be using any and all problems discovered to improve our testing matrix.

Thank you again to everyone for your patience while we work on this. Have a great weekend and stay safe.

kphillips

@nick108 Please open a ticket with our support team. If you have truly hit a bug, we'd like to know about it so that we can make sure any revised release includes it.

alpharulez

@kphillips Thank you. Are you in a position to be able to share an ETA?