21.02 Sudden lockup

kphillips

@alpharulez Because the repos for 21.02 were pulled to keep people from upgrading any more until we release a fix, packages are also affected. You'll need to revert to 2.4.5p1 to be able to install packages again (make sure you set the "Previous Stable Release" in your Update settings before trying).

kphillips

@rloeb If you are absolutely stuck and need to remain on 21.02, you can do the following:

Go to Diagnostics --> Command Prompt and run "echo hw.ncpu=1 >> /boot/loader.conf.local" without quotes.

Reboot.

This will halve your firewall's performance because it artificially limits your CPU to one core, but it also gets rid of the crashing.

However, if you can revert to 2.4.5p1 DO THAT. And if you do the above, don't forget to remove it later unless you like having your firewall be artificially limited in performance for the rest of it's life.

rloeb

Thank you!

rloeb

@kphillips FYI. Went to 1 CPU. Snort does not start. Snort does not appear in Services menu. Re-installing snort fails; Window just sits there. Would like to have that protection, but not sure what to do next.

lnguyen

@kphillips said in 21.02 Sudden lockup:

Command Prompt and run "echo hw.ncpu=1 >> /boot/loader.conf" without quotes.

I think what @jimp stated was:
Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

echo hw.ncpu=1 >> /boot/loader.conf.local

I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:

rm /boot/loader.conf.local

rloeb

@lnguyen Nice catch. The command did not create the file. My Linux is pretty feeble these days, so I'm unclear what to do next. I'll see if I can pull one of my techies off what they're doing and he can chase this. I'll go back to running the company, which is all I'm competent to do.

kphillips

@lnguyen You are correct. I've updated my original post.

kphillips

@rloeb Snort is broken on the SG-3100 and pfSense Plus 21.02. This is due to a bug in the package, not pfSense Plus. Snort has some badly coded components that Intel CPUs usually just "auto fix", but on ARM that mechanism doesn't exist. As such, something appears to have broken during the move to 21.02. We have a bug report for that, but if you need snort you'll want to be on 2.4.5p1 until that is sorted. Otherwise, Suricata works fine on the SG-3100 AFAIK right now on 21.02.

rloeb

@kphillips Thank you. Good to know. Going to try to tough it out. Alternative is to switch to a gateway router with no filtering, just to keep folks productive.

styxl

@kphillips did they block all packages from 2.4.5-p1? i rolled back and restored from 21.02 now i cannot re-install all the packages for 2.4.5-p1 what is going on?

SteveITS

@styxl Did you set System/Update/Update Settings to "previous stable version (2.4.5)"? I can see packages on a 3100 that wasn't upgraded.
You might try https://docs.netgate.com/pfsense/en/latest/troubleshooting/pkg-broken-database.html.

kphillips

@styxl Make sure you select "Previous Stable Version (2.4.x)" under System --> Update. The repos for 21.02 are now offline to keep people from upgrading to it for the SG-3100 right now.

styxl

@teamits i just did and it worked, thx

alpharulez

Does anyone know if Suricata on 21.02 is impacted the same as Snort? Thanks!

stephenw10

On the SG-3100 it would be, in blocking mode at least. Like Snort it has to reload the ruleset whenever a new IP is added to the block table.

Steve

alpharulez

@stephenw10 ok thanks for the response
Will hold fire.

stephenw10

....unless you're seeing this: https://redmine.pfsense.org/issues/11466
That applies to Snort only.

router

Hello, is there an update on this issue?

I'm experiencing major packet loss and unable to download new packages.

I've already added hw.ncpu=1 to /boot/loader.conf.local.
This had no noticeable affect.

Our systems are completely degraded by this issue. We cannot handle the risk and downtime required to reinstall. This is a major impact for us.

Thanks...

kphillips

@router Packet loss is not a symptom of this issue. The SG-3100 would completely freeze up and force a reboot. If you have packet loss, its not 21.02 most likely. Check your gateway monitoring.

bldnightowl

Well, add me to the list of people that downgraded back to 2.4.5p1. And that was quite a hassle/nightmare by itself. Even with pfBlockerNG removed (which was an unsustainable solution for any period of time, of course), the system was still freezing or behaving erratically at times. Getting the packages back to the way they were pre-21.02 did not automatically happen as it should have -- and I had to manually intervene several times. This failed upgrade cost me a couple of days at least of my time, and like others I am very unhappy about that.

I am a software engineer too and understand how very hard it is to test field configurations for an extremely customizable product. So I'm not trying to make anyone at Netgate fill badly --- but this was pretty disastrous for many users, and a a detailed post mortem explaining what went wrong, why and how it will be avoided in the future would be hugely appreciated. For example, it appears your QA did not have pfBlockerNG(-devel) (which I would be willing to guess is in very widespread use) properly in its standard performance testsuite. I hope that's been rectified.

Thanks for the hard work and responsiveness when things did blow up, particularly you moderators on the front lines absorbing all the screams from your users. And especially to those of you responding while impacted by the much worse disasters in Texas.