pfSense Plus and SG-3100
-
A problem has been reported by some users of the Netgate SG-3100 appliance who have upgraded to pfSense Plus version 21.02. Our engineering team is working to correct the issue as quickly as possible. In the meantime, we have suspend the upgrade for the SG-3100 and SG-1000 (as precaution). We expect to provide a solution to the issue, which appears to be related to reloading the packet filter, as soon as testing is complete. We apologize for the inconvenience.
-
@stephenw10 , I suggest users to downgrade.
-
@stephenw10 Hello, when do we expect to have the fix release? thanks!
-
As soon as possible. Hard to say anything more at this point.
It's a lot closer now that we have replicated it locally though.Steve
-
@stephenw10 If you want me to test something, just tell me, I'm currently using the 21.02 in a SG-3100 and I would be glad to be able to help you folks..
-
Thanks for the offer. I'll reach out if we need more data points.
It's in the hands of our developers now though and they are able to replicate it on demand.Steve
-
@mcury same for me!
-
Is there an email alert we could get to know when this is fixed? (I'll poke around and see if I can find a way to sign up for an alert in the meantime).
We have 2 SG-3100's and 1 was upgraded.
-
I have the same problem and I am not able to down grade as the package manager seems to be broken on the SG3100.
I need to get this stabilized for my customer. As a workaround, I am stating to build up a 'home edition' one just to get him stable.
Would I be correct in assuming that if I could roll back, I would not be having these random/every few hour/ halts?
Like others, I have been monitoring the Serial port, but very little information is available to move forward to resolving this.
pkg update -f
Updating pfSense-core repository catalogue...
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-core/meta.txz: Not Found
repository pfSense-core has no meta file, using default settings
pkg: https://files01.netgate.com/pkg/pfSense_plus-v21_02_armv7-core/packagesite.txz: Not Found
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: https://files00.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02/meta.txz: Not Found
repository pfSense has no meta file, using default settings
pkg: https://files00.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02/packagesite.txz: Not Found
Unable to update repository pfSense
Error updating repositories!So, this likely should be a new thread. But, I checked
cat /usr/local/share/pfSense/pkg/repos/pfSense-repo.conf
FreeBSD: { enabled: no }pfSense-core: {
url: "pkg+https://firmware.netgate.com/pkg/pfSense_plus-v21_02_armv7-core",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/local/share/pfSense/keys/pkg",
enabled: yes
}pfSense: {
url: "pkg+https://firmware.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/local/share/pfSense/keys/pkg",
enabled: yes
}And
https://firmware.netgate.com/pkg/pfSense_plus-v21_02_armv7-pfSense_plus-v21_02
Does not exist at firmware.netgate.com
thanks in advance,
-
@va3mw you'll need to backup your config and re-install pfSense 2.4.5. See https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html
-
Ahh, I've been experiencing the "wonkiness" today, so I'm glad it's not just me.
If someone has already upgraded, and doesn't really want to downgrade, is there any other workaround, or is it basically just restarting the firewall to get it operational again?
-
You can disable one CPU core and it will avoid ever hitting the lock. Obviously performance will be reduced but that may not be an issue for a lot of use cases. Run:
echo hw.ncpu=1 >> /boot/loader.conf.localThen reboot.
Remove or comment out that line later after this is fixed.
Steve
-
@stephenw10 said in pfSense Plus and SG-3100:
echo hw.ncpu=1 >> /boot/loader.conf.local
Done and done. Thank you Steve!
Will updates on the update be placed in this thread (that I'm watching)?
-
Yes, this thread will be updated.
-
@amarand said in pfSense Plus and SG-3100:
@stephenw10 said in pfSense Plus and SG-3100:
echo hw.ncpu=1 >> /boot/loader.conf.local
Done and done. Thank you Steve!
Will updates on the update be placed in this thread (that I'm watching)?
Did this work for you? Im stuck rebooting randomly during the day also.
-
Just added this entry to the file a few minutes ago, so we'll see!
I only had a single failure requiring a reboot during the day today...so I was hoping to avoid experiencing that again.
Glad there's a quick and easy workaround.
-
@mr_aj So far so good for me, 3 hrs in on a single cpu.
-
@oldmanniko said in pfSense Plus and SG-3100:
@mr_aj So far so good for me, 3 hrs in on a single cpu.
All is well with this quick fix. Thanks!
-
Good to hear.
We think we have found the root cause of this and will be testing fixes imminently.
Technical details here for those who may be interested:
https://reviews.freebsd.org/D28821Steve
-
@artooro Thanks. That is what I did.
-
@stephenw10
Is there a possibility to 'upgrade´ from the 21.02 to the new one without having to downgrade first? I am experiencing the problem, as in not being able to retrieve any packages. Or is the only way to do a downgrade first and then upgrade later? Can I export my settings from my current 21.02 one and import them again when I upgrade?Thanks for your guys work on this. Much appreciated
-
Yes, I expect it to be 21.02_1 or similar when made available. You should be able to upgrade to it from either 2.4.5p1 or 21.02.
Yes, the config version will be the same but you can always import an older config into a newer pfSense version anyway. You will be able to here.Steve
-
@stephenw10
Does that mean that the url mentioned earlier in this thread will be up and running? -
The pkg server? Yes, it will be. When it's available the update should show on the dashboard like any previous update.
Steve
-
@stephenw10 said in pfSense Plus and SG-3100:
You can disable one CPU core and it will avoid ever hitting the lock. Obviously performance will be reduced but that may not be an issue for a lot of use cases. Run:
echo hw.ncpu=1 >> /boot/loader.conf.localThen reboot.
Remove or comment out that line later after this is fixed.
Steve
i went to diagnostics > command prompt and ran the command. got the green success screen i guess you call it and then rebooted
after a reboot when i go to diagnostics > edit file > open loader.conf nothing has changed.
do you have to use the console or SSH for this to complete? -
It doesn't add it to loader.conf which might get overwritten.
/boot/loader.conf.local
-
I really appreciate the effort to root cause this, and for the easy workaround -- thank you!
What is the typical turn-around for pushing out a hotfix like this, if nothing goes wrong during testing? I've been holding-off on downgrading to 2.4.5 thinking a fix may be landing soon.
I put in the ncpu work-around after the bug started disrupting work meetings and online schooling, but it does cause the WAN bandwidth to max out at ~650Mbps.
-
Very soon.
We are testing new images now. I've been hammering the SG-3100 with traffic that easily triggered this before and it seems solid so far.Steve
-
New version for SG-3100 is out and I see it available for download. Will be happy to have multicore back.
-
Curious if anyone has attempted the update yet. Any results?
-
The update "bricked" (strong word - trying to get in via serial console right now - all three lights flashing ominously on the front), so please be careful installing this update unless you have your serial console cable ready and a few hours to troubleshoot.
My update started an hour ago, and I'm just now getting things set-up to see what's wrong.
Buyer beware. Caveat emptor. YMMV. I'm connected directly to my cable modem and hopefully that won't be the case for much longer.
Monitoring this thread to see if anyone else has the issue with the hotfix.
-
@nokkief I'm a glutton for punishment I suppose. I installed the new version 2 hrs ago. No issues yet.
-
@nokkief yes, and the system was unresponsive with blue lights pulsing on front until I power cycled after about an hour.
Logs indicate it processed the patch and initiated a reboot but seems to never actually rebooted.Seems ok after the power cycle
-
I didn't try yet, opened a ticket at go.netgate.com to request the firmware, but it's not available yet.
The only available path to p1 is through the upgrade mechanism on your firewall at this time..I want to perform a clean install.
After installing, and setting up interfaces and switch confiig, I'll restore aliases, firewall rules, dhcp mappings, all from my previous xml saved config.
After that, manually set up the certs and configure acme, pfblocker and etc.. -
The 40+ year Unix administrator in me decided to get the serial console hooked up before doing anything else. It was responding/repeating characters, but that's it. Figured after an hour of blinky-flashing lights, it was probably safe to power cycle. I worry a lot about power cycling in the middle of an update...can actually ruin/brick things, especially if an EEPROM is being written to, or whatever.
But yes, power cycled with the console connected, watched it boot, came up fine.
I need to remind myself NOT to do this upgrades during the middle of the work day.
Netgear support: I created a ticket via email. If you'd like to check my router's logs to find out what happened, I'd be more than happy to open things up for you to investigate. Sounds like I wasn't the only person with a "blinking light" issue post-install.
-
Also, I just want to say "thank you Netgate!" for including a working/tested serial console cable inside the SG-3100 box.
I had given away ALL of my Mini-USB cables and was frantically searching for one (out of hundreds of cables), and then I thought to check the box. Whew!
-
Also, for folks who used this workaround, don't forget to remove the "hw.ncpu=1" entry in your /boot/loader.conf.local file after successfully upgrading to the hotfix version.
-
Hey @stephenw10, are you able to confirm that NEW SG-3100's are now being shipped with the updated image? I just ordered one last week that shipped yesterday afternoon, so I'm hoping it will have the fix already applied.
-
requested the latest image file, restored that, restored my backup. everything works great
-
@nokkief I updated as soon as it became available. Updated through the UI and have had no issues since. Im just one data point for you.
