Upgrade to 21.02-RELEASE borked on SG-3100

[yaminb]

@yaminb
Should update myself. Looks like my sg 3100 just crashed as well after about 1 day. reboot seems to have it back in order.

I'll try the 1 cpu temporary fix

[ahking19]

@lohphat if you have a SG-3100 don't upgrade until a fix is available. Stay on 2.4.5-p1.

[yaminb]

@jimp said in Upgrade to 21.02-RELEASE borked on SG-3100:

hw.ncpu=1

Really silly question just for completeness.

I don't have a loader.conf.local.

I made the change in loader.conf. I'm assuming this is good.

/boot: cat loader.conf 
kern.shutdown.secure_halt=1
kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"
hw.ncpu=1
boot_serial="YES"
console="comconsole"
comconsole_speed="115200"
hw.e6000sw.default_disabled=1
autoboot_delay="3"
hw.hn.vf_transparent="0"
hw.hn.use_if_start="1"

[jimp]

Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

@lohphat said in Upgrade to 21.02-RELEASE borked on SG-3100:

what the release QA process is and how this may have slipped through testing.

Several of us have 3100s and use them in various ways, including a couple of us using them on the edge, running snapshots, but this problem takes a specific load and setup to trigger that apparently none of us hit somehow. Usually dogfooding the snapshots catches most things, but there are many more real-world configurations than we can possibly test.

[rsherwood_va]

@jimp said in Upgrade to 21.02-RELEASE borked on SG-3100:

Usually dogfooding the snapshots catches most things, but there are many more real-world configurations than we can possibly test.

I don't think my config is all that exotic, but should I share it with you all? Do you know what the issue is?

[jimp]

We found a way to reliably trigger it here in lab conditions so we can work on it, no need to provide more info at the moment.

[lohphat]

@ahking19 Clearly. I'm usually don't mind being a testing guinea pig but not this week, too much going on.

[Meizel]

Hi All,

Yesterday dune the update and you can tell, also for me not good.
My SG-3100 is stuck and keeps hanging in to A boot loop....

With the serial connected, I can see that the Marvell U-Boot is OK.
When it start to boot pfsense i got this error.

login-to-view

So need to create A ticket for reinstalling pfsense.
So my network needs to rely on mu ubnt ER-8 pro.

Greetings Dennis

[EvaKnievel]

Ran into an interesting problem.

I can't add back my packages after moving back to an earlier version and I think it's because the place where it's checking for packages is the old place?

Here is the error

[2.4.5-RELEASE][admin@pfSense.i.lacy.ie]/root: pkg search FreeRadius
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
pkg: Repository pfSense-core missing. 'pkg update' required
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-core/meta.txz: Not Found
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-core/packagesite.txz: Not Found
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02/meta.txz: Not Found
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02/packagesite.txz: Not Found

[bmeeks]

The update to 21.02 for SG-3100 appliances has been temporarily pulled to fix a bug. So that has probably confused the package system as well on those appliances.

If you can, just be patient for a day or two and Netgate should get a new update for the SG-3100 posted. If you can't wait, there are instructions for rolling back to the previous release. You will need to open a ticket with Netgate for that. There is no charge for providing you a rollback image, and they will send you the link via email shortly after you open a ticket. Start the process here: https://go.netgate.com/support/login.

[lnguyen]

@evaknievel

Under System | Update, set to Previous 2.4.5 to get the correct package repos for 2.4.5-p1.

[Meizel]

After my ticket for support.

Within two minutes got the answer from the netgate team, gread work for that ;)
within 10 minutes my SG-310 was running again.

Only i going to wait A few days to put it back in production.
Mainly because of testing this upgrade and their are still no packages available.
Its what Bill said "give Netgate the time to work it out"

[solarizde]

@evaknievel Sometimes if you switch the branch back it may the case that PKG itself gets stuck. A fix is in the prev branche to issue "pkg-static install -f pkg"

Small write up about that problem often Showing Shared object "libarchive.so.7" not found, required by "pkg"

For my SG-3100 it's strangly fine now. No crash since >24h but need to say I disabled all packages. Hope there is a "fix" soonish don't want to roll back completely.

[DeltaOne]

@yaminb said in Upgrade to 21.02-RELEASE borked on SG-3100:

@jimp said in Upgrade to 21.02-RELEASE borked on SG-3100:

hw.ncpu=1

Really silly question just for completeness.

I don't have a loader.conf.local.

I made the change in loader.conf. I'm assuming this is good.

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

[WarlordZico]

@warlordzico said in Upgrade to 21.02-RELEASE borked on SG-3100:

I've upgraded my SG-3100 yesterday evening. I never had a respons from the SG-3100 anymore.
This morning i saw three blue leds slowly blinking in sync.
I pulled the power plug. The firewall restarted and finnished the setup. Till now, everything works with only one remark. Snort doesn't startup anymore. Tonight i'm gonna remove the snort package and do a clean install of it.

Running 21.02 without any package installed, no hickups till now.
Everything works fine except: i cann't install any package. Package manager has following error: unable to retrieve package information.

Next step, trying to get a Wireguard VPN running.

[EvaKnievel]

@lnguyen perfect thank you!

[lnguyen]

@deltaone said in Upgrade to 21.02-RELEASE borked on SG-3100:

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

I think what @jimp stated was:
Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

echo hw.ncpu=1 >> /boot/loader.conf.local

I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:

rm /boot/loader.conf.local

[DeltaOne]

@lnguyen said in Upgrade to 21.02-RELEASE borked on SG-3100:

@deltaone said in Upgrade to 21.02-RELEASE borked on SG-3100:

Our 3100 locked up after about 16 hours. Then twice more in the next 8 hours. We applied the "hw.ncpu=1" fix to loader.conf and it has been running fine for the last 8 hours. (Like @yaminb we could not find a loader.conf.local.)

I think what @jimp stated was:
Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

echo hw.ncpu=1 >> /boot/loader.conf.local

I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:

rm /boot/loader.conf.local

Good catch. In our case, we do want loader.conf to be over-written so we are again back at two CPUs.

[lnguyen]

@deltaone Just issue the command above and reboot. Once an official patch is released, you can issue the rm command and reboot.

[behemyth]

@jimp

If you make the loader.conf.local file, does the appliance use it since its the last one listed in the string?

loader_conf_files="/boot/device.hints /boot/loader.conf /boot/loader.conf.local"