Upgrade to 21.02-RELEASE borked on SG-3100

Meizel · Feb 20, 2021, 12:49 AM

After my ticket for support.

Within two minutes got the answer from the netgate team, gread work for that ;)
within 10 minutes my SG-310 was running again.

Only i going to wait A few days to put it back in production.
Mainly because of testing this upgrade and their are still no packages available.
Its what Bill said "give Netgate the time to work it out"

solarizde · Feb 20, 2021, 1:54 PM

@evaknievel Sometimes if you switch the branch back it may the case that PKG itself gets stuck. A fix is in the prev branche to issue "pkg-static install -f pkg"

Small write up about that problem often Showing Shared object "libarchive.so.7" not found, required by "pkg"

For my SG-3100 it's strangly fine now. No crash since >24h but need to say I disabled all packages. Hope there is a "fix" soonish don't want to roll back completely.

DeltaOne · Feb 20, 2021, 2:05 PM

@yaminb said in Upgrade to 21.02-RELEASE borked on SG-3100:

@jimp said in Upgrade to 21.02-RELEASE borked on SG-3100:

hw.ncpu=1

Really silly question just for completeness.

I don't have a loader.conf.local.

I made the change in loader.conf. I'm assuming this is good.

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

WarlordZico · Feb 20, 2021, 2:41 PM

@warlordzico said in Upgrade to 21.02-RELEASE borked on SG-3100:

I've upgraded my SG-3100 yesterday evening. I never had a respons from the SG-3100 anymore.
This morning i saw three blue leds slowly blinking in sync.
I pulled the power plug. The firewall restarted and finnished the setup. Till now, everything works with only one remark. Snort doesn't startup anymore. Tonight i'm gonna remove the snort package and do a clean install of it.

Running 21.02 without any package installed, no hickups till now.
Everything works fine except: i cann't install any package. Package manager has following error: unable to retrieve package information.

Next step, trying to get a Wireguard VPN running.

EvaKnievel · Feb 20, 2021, 2:56 PM

@lnguyen perfect thank you!

lnguyen · Feb 20, 2021, 4:39 PM

@deltaone said in Upgrade to 21.02-RELEASE borked on SG-3100:

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

I think what @jimp stated was:
Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

echo hw.ncpu=1 >> /boot/loader.conf.local

I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:

rm /boot/loader.conf.local

DeltaOne · Feb 20, 2021, 4:48 PM

@lnguyen said in Upgrade to 21.02-RELEASE borked on SG-3100:

@deltaone said in Upgrade to 21.02-RELEASE borked on SG-3100:

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

I think what @jimp stated was:
Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.
echo hw.ncpu=1 >> /boot/loader.conf.local
I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:
rm /boot/loader.conf.local

Good catch. In our case, we do want loader.conf to be over-written so we are again back at two CPUs.

lnguyen · Feb 20, 2021, 4:56 PM

@deltaone Just issue the command above and reboot. Once an official patch is released, you can issue the rm command and reboot.

behemyth · Feb 20, 2021, 5:44 PM

@jimp

If you make the loader.conf.local file, does the appliance use it since its the last one listed in the string?

loader_conf_files="/boot/device.hints /boot/loader.conf /boot/loader.conf.local"

DeltaOne · Feb 20, 2021, 6:13 PM

@lnguyen said in Upgrade to 21.02-RELEASE borked on SG-3100:

@deltaone Just issue the command above and reboot. Once an official patch is released, you can issue the rm command and reboot.

Have done so. Thanks.

rsherwood_va · Feb 23, 2021, 4:42 PM

@rsherwood_va I have been running smoothly since the original problem. I just had a lockup this morning - tried creating /boot/loader.conf.local and adding hw.ncpu=1, then rebooting as recommended by the team here, but that did not resolve the issue. As before, the following process worked:

ssh in to the router (you may need to use the IP address)
choose option 11 (Restart webConfigurator)
log in to the web gui
(optional - not sure if this helped) Under status -> services, start the DNS and DHCPD services
Under system-> setup wizard. Accept all the answers (it will remember what you chose last time, except for the admin password - reenter the old one)

rerunning the PFSense first-time setup wizard (accepting all the previously chosen values and choosing to enter the existing admin password as the "new" admin password) resolved the issue - I am now running smoothly.

I haven't seen the new update yet, so I'm assuming the issue is trickier than they thought. If you are stuck, try running the wizard again.

yaminb · Feb 24, 2021, 2:25 PM

Another update.

With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
It feels like ncpu=1 has helped, but I don't think it's the issue

I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

bmeeks · Feb 24, 2021, 7:55 PM

@yaminb said in Upgrade to 21.02-RELEASE borked on SG-3100:

Another update.

With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
It feels like ncpu=1 has helped, but I don't think it's the issue

I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

The switch to using a single CPU is a workaround that minimizes the chance of hitting the bug, but it does not eliminate the chance.

The actual problem has been identified and a fix is being tested. Here is a link to the discussion by the FreeBSD kernel programming nerds* of the problem and the fix: https://reviews.freebsd.org/D28821. I believe the pfSense team is now vigorously testing images with this fix applied to be sure the fix is really "the fix". And from the activity on the Redmine bug site for pfSense, it looks like a few other bugs are being addressed as well.

Note -- I don't mean "nerds" in an insulting sense . But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

RobertBrooks · Feb 25, 2021, 4:41 AM

How likely is this issue specific to the SG-3100?

My faithful SG-2400 has required a few reboots since the upgrade,
after never needing to be restarted in its five years of service.

                         -- Rob

bmeeks · Feb 25, 2021, 4:45 AM

@robertbrooks said in Upgrade to 21.02-RELEASE borked on SG-3100:

How likely is this issue specific to the SG-3100?

My faithful SG-2400 has required a few reboots since the upgrade,
after never needing to be restarted in its five years of service.
                         -- Rob

An SG-2400? Do you maybe mean the SG-2100? If 2100, that is also an ARM CPU, but it is a 64-bit version.

RobertBrooks · Feb 25, 2021, 4:50 AM

@bmeeks

I misspoke; I meant an SG-2440, which I recognize has an Atom CPU; I didn't know if this problem is specific to the
ARM.

                                        -- Rob

lohphat · Feb 25, 2021, 5:48 AM

@bmeeks said in Upgrade to 21.02-RELEASE borked on SG-3100:

Note -- I don't mean "nerds" in an insulting sense . But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

I've worked with kernel and compiler hackers and they are indeed a breed apart. You have to be able to abstract activity across interrupts, CPU rings, and microcode idiosyncrasies. It's the difference between playing checkers and 4D chess.

lohphat · Feb 26, 2021, 7:54 AM

Note there is now a 21.02_1 release for SG-3100 to fix the FreeBSD kernel bug causing the instability.

https://forum.netgate.com/topic/161421/pfsense-plus-21-02-p1-now-available/11

DeltaOne · Feb 27, 2021, 3:09 PM

@lohphat

We applied the 21.02_1 update about 12. hours ago. So far...all is good, our 3100 is running normally and back on two CPUs.

Many thanks to the Netgate teams for this quick fix!

JGdgZPQatDDjpA · Feb 26, 2021, 11:55 PM

@jgdgzpqatddjpa

Just upgraded my 3100. No issues so far.