Upgrade to 21.02-RELEASE borked on SG-3100

[softcoder]

Same here SG-3100 upgraded to 21.02-RELEASE (arm)

After a while throughout today the GUI was unresponsive, both ethernet AND wireless clients are unable to even ping pfsense router ip address, all LED's look normal and after powering off the router and it comes back all is working 3-4 times today already.

Does anyone know how to force a downgrade back to 2.4?

[geeklex]

@sabennett Having similar problems with a 3100. Thanks for opening a ticket.

[jsnaza]

@softcoder Information on a downgrade is here:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

[ElGuapo]

I have the same issue with my SG-3100:
-upgraded fine to 21.02 and all look good at first then it hang/froze no DHCP no DNS and no response from the GUI the only way to get it back was to unplug it and plug it back, then it froze again after 1 hr then again after 3 hrs (just normal usage) there is no info on the System Logs that points to what hang it just register the bootup

I will downgrade too.

[solarizde]

All this Update makes me a bit frustrated. It's one point why I choose to buy original netgate appliances instead of build your own. I hoped at least for their own build Products they do a very intense test before rolling out a backup.

No Question you cant forseen everything but if you read through the forum how many have trouble it seems like a bad release / quality control.

Hope they will find the issue and take some learnings.

For my upgraded testing SG-3100 it's not important, it is a non live LAB system but until this one is rock solid stable I will not start rolling out CPE Upgrades :)

[Kuser]

@solarizde My system has now been running for 21h after failing about 10x at random 5m-6h intervals. Probably not related, but the only difference is that USB console is connected for debug purposes...

[rabbiwan]

Hello,
Yesterday, I have re-installedmy device SG3100 with the firmware 21.02.
I have contact the support and them wend me the image to flash the device.
I have doing the configuration. During 2 hours no problem but after the system has explose and all the configuration has down like the other time....
I will try to downgrade the version...

[shadtheman]

Same problem here, SG-3100 (21.02) runs for anything up to a few hours and then just locks up completely. Hard reset required, logs do not yield any clues at this stage. I've only got two packages installed PfBlockerNG and BandwidthD. Will try another clean install but not very hopeful as others have found this has not solved the problem.
Also just wondering if this problem is primarily occurring with the SG-3100.

[Kuser]

I have two devices available. First experienced problems right away, the other remote install just crashed/became unresponsible after ~1.5 day.

After power cycle openvpn now works, but stopped routing internal subnets/forwarding traffic. Unable to connect to gui.

[geeklex]

Netgate just updated their twitter account with this.

Netgate
@NetgateUSA

A problem has been reported by some users of the Netgate SG-3100 appliance who have upgraded to pfSense Plus version 21.02. Our engineering team is working to correct the issue as quickly as possible. In the meantime, we have suspend the upgrade for the SG-3100.

[jimp]

As a temporary workaround until we can put out a fix, you can reduce the number of CPUs used by the OS to 1 by adding hw.ncpu=1 to /boot/loader.conf.local and then rebooting. You'll lose some performance but it appears (so far) to not trigger the issue when set that way. Otherwise, you can step back to 2.4.5-p1 and wait for a fix.

[EvaKnievel]

@jimp How do we rollback to the previous version?

[jsnaza]

@evaknievel Information on how to downgrade is here:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

[EvaKnievel]

@jsnaza we are talking minimum 2 days for a fix right?

[jsnaza]

@evaknievel I am not sure I am just another person that ran into this issue and downgraded. I am not one of the developers. I am just sharing the knowledge I have.

[lohphat]

@jimp Thank you.

What may also be helpful is to inform the user base on what the release QA process is and how this may have slipped through testing.

Infrastructure component updates are highly sensitive (as you're well aware) and thus confidence in quality is key to keeping "customers" confident in the product.

Shit happens but there should be a post-mortem to figure out how this slipped by.

[EvaKnievel]

@jimp we are talking minimum 2 days for this fix right?

[lohphat]

@jimp If we make the single CPU change in 2.4.5-p1 before we install the update, will that setting persist after the initial reboot into 21.02?

[yaminb]

@yaminb
Should update myself. Looks like my sg 3100 just crashed as well after about 1 day. reboot seems to have it back in order.

I'll try the 1 cpu temporary fix

[ahking19]

@lohphat if you have a SG-3100 don't upgrade until a fix is available. Stay on 2.4.5-p1.