pfSense Plus and SG-3100

Stormdude · Feb 20, 2021, 9:52 PM

@stephenw10 Any specifics on the symptoms of the issue? Since the upgrade, my SG-3100 becomes unresponsive every day or two and requires power cycle to become usable again.

stephenw10 · Feb 21, 2021, 12:22 AM

Pretty much exactly that see: https://redmine.pfsense.org/issues/11444

As a temporary workaround you can disable one CPU core, that prevents the lock being hit.
At the command line run:

echo hw.ncpu=1 >> /boot/loader.conf.local

Then reboot.
You can removed that file or comment out the line later when a fix is available.

We think we now have the cause of this and will be testing fixes very soon.
https://reviews.freebsd.org/D28821

Steve

sigtom · Feb 22, 2021, 4:50 AM

I saw the same behavior on my SG-3100. Ive downgraded to 2.4.5p1 for now, as not only hard hard reboots required when it locks up, but there are no packages available on this device at this time for the 21.02 branch. I suggest downgrading, and waiting till there is a fixed, stable update released. Im sure Netgate/pFsense will resolve this quickly for us on the SG-3100/1000 platform soon.

mogarchy · Feb 22, 2021, 11:00 PM

I had the same behavior, ended up wiping and installing from scratch, but has already locked up once since doing that as well.

Now I don't only have a system that's prone to crashing, but I cannot install any packages either. SG-3100 has to be one of their most popular devices, how did this make it out of testing?!

stephenw10 · Feb 22, 2021, 11:17 PM

In 2.4.5p1? If so that's something unrelated.

If you re-installed 21.02 you will hit this if you have high traffic and something is reloading the filter unless you apply the single CPU core workaround.

We are testing the fixed images now.

Steve

behemyth · Feb 23, 2021, 4:51 AM

@stephenw10

I’m running the newest 21.05 images and the rule counters are broken and in this latest update it looks like UPnP doesn’t work anymore. Have you guys experienced this?

I guess this only matters if that’s the images you guys are testing.

flsnowbird · Feb 22, 2021, 11:28 PM

I'm running fine now on 21.02 with the 1 cpu fix mentioned above. Only issue I see is slower throughput on my wireguard connection. No lockups for 27 hours so far.

stephenw10 · Feb 23, 2021, 4:51 AM

@behemyth Yes 21.05 is the equivalent of 2.6. It's head effectively, anything could be broken!

The images we are testing are purely for this fix. Looking good so far.

Steve

beachbum2021 · Feb 24, 2021, 12:46 AM

@stephenw10 some clarity if you do not mind. The fix isnt a fix, it's a workaround, correct. When will the real fix be completed...a week..a month etc.

stephenw10 · Feb 24, 2021, 2:01 PM

No, it's a real fix. See: https://reviews.freebsd.org/D28821

beachbum2021 · Feb 25, 2021, 2:09 AM

@stephenw10 said in pfSense Plus and SG-3100:

https://reviews.freebsd.org/D28821

i guess i'm just a little confused then. Is the fix going to degrade performance if we're limiting the hardware cpu cores? Do we need to reapproach our load on a 3100 with these changes?

stephenw10 · Feb 26, 2021, 1:44 AM

No, the temporary workaround was to disable one CPU core. That is not required with the fix that is now in 21.02p1.

Steve

flsnowbird · Feb 26, 2021, 3:49 AM

@stephenw10 Do we remove that line from the config before or after applying the patch? Want the update to go as flawless as possible.

bcruze · Feb 26, 2021, 8:52 AM

@flsnowbird said in pfSense Plus and SG-3100:

@stephenw10 Do we remove that line from the config before or after applying the patch? Want the update to go as flawless as possible.

I did it before
Then did the update, then rebooted the firewall so the limitation were not in place

stephenw10 · Feb 26, 2021, 2:44 PM

Yes, you could do either but you will need to reboot after removing it since that is applied as a loader variable at boot.

Steve

fg · Feb 26, 2021, 11:07 PM

@stephenw10 It would be nice if you could explain in any detail what the problem is/was. I just purchased an SG3100 and I got no emailed warning about any potential issue either (would be nice really).

That said I answered the upgrade survey not knowing all that was meant by the upgrade to pfsense+. If you're going to make a really good firewall product in terms of control panel / GUI and work process you should get your hands on Microsoft's ISA firewall from some time ago. I don't know why the product disappeared but it was the best gateway/firewall/router I've ever used. I looked for a reasonable replacement and found pfsense to be the next best thing but you guys still don't measure up to something from 15 or so years ago. ;0)

Now that you're on a serious commercial product path that is what I'd look from you guys.

SteveITS · Feb 26, 2021, 11:50 PM

@fg said in pfSense Plus and SG-3100:

explain in any detail what the problem is/was

https://www.netgate.com/blog/pfsense-obscure-bugs-and-code-wizards.html

rloeb · Feb 27, 2021, 12:03 AM

@teamits Wonderful blog explanation. Been a long time since I debugged code, and I enjoyed the description of the "hunt." Kudos to the Netgate team for a superb job.

flsnowbird · Feb 27, 2021, 10:33 PM

This same bug was in 2.5 DEVEL and I reported it to support, but was only told to revert to 2.4 stable. I guess I was just too exited about wireguard. Next big upgrade I'm waiting a few weeks before jumping on it.