pfSense Plus and SG-3100

flsnowbird

@sdd @NokkieF I upgraded from 21.02 to 21.02_1 and had the 3 blinking lights.

Amarand

@nokkief

I upgraded to 21.02 with no issues.

I then experienced the hanging, so I added the CPU limiter workaround.

The -p1 hotfix released, and I applied it from the web-site.

The upgrade log showed, in the web-browser, "success" at the end, no errors.

After five minutes of "retying, please wait" and no SSH response, I went downstairs to see the blinking lights.

I forgot about the console/cable, and opened a ticket with Netgate because I wasn't sure if I should unplug a system that was potentially in the process of upgrading. That's a good way to actually brick any appliance/embedded system.

After figuring out that the console was the safest way forward, got that hooked up, saw no response on the console, and decided a power cycle was warranted. That was almost an hour from upgrade.

-p1 came back up just fine.

Amarand

@sdd said in pfSense Plus and SG-3100:

+1

I'd like to know too -- I need this fix, but watching this thread has convinced me that I should request the updated USB image, and block out an hour during "off-peak" usage to attempt the upgrade.

Installing "off-peak" is always a good idea anyway.

If folks upgrading from 21.02 aren't having any issues, I might roll the dice.

If you're physically there by the firewall, go ahead and apply it, as long as you can afford a short outage. But I would have a serial console hooked up and watching BEFORE you apply. This way, you can see where it's hanging, if it hangs.

As an aside, for me, it applied -p1 from 21.02, hung, I power cycled, it came up on -p1 hotfix. It shouldn't have locked up post-install, but it did successfully update.

Amarand

@flsnowbird

Oh, and, of course, if you have folks at home who are willing to trace that power cable from the back of the unit, to where it plugs into the wall, that might be easier to unplug, wait a few seconds, then plug it back in. That's slightly easier than unscrewing the screw.

I already have my kid trained to detect cable modem issues based on LED statuses. He knows how to unplug the cable modem and the main house networking switch, watch for things to come back up, and knows how to report that back to me.

Might be good opportunity to train your family!

Amarand

@amarand said in pfSense Plus and SG-3100:

@sdd said in pfSense Plus and SG-3100:

+1

I'd like to know too -- I need this fix, but watching this thread has convinced me that I should request the updated USB image, and block out an hour during "off-peak" usage to attempt the upgrade.

Installing "off-peak" is always a good idea anyway.

If folks upgrading from 21.02 aren't having any issues, I might roll the dice.

If you're physically there by the firewall, go ahead and apply it, as long as you can afford a short outage. But I would have a serial console hooked up and watching BEFORE you apply. This way, you can see where it's hanging, if it hangs.

As an aside, for me, it applied -p1 from 21.02, hung, I power cycled, it came up on -p1 hotfix. It shouldn't have locked up post-install, but it did successfully update.

In my line of work, back in the 90's, I "bricked" a super computer I was installing, and had to wait for someone to bring me a flash image on USB. That was MY stupidity.

I think this is just some weird post-install glitch Netgate hasn't figured out yet. If someone who's on 21.02 is willing to hook up a serial console pre-upgrade to -p1, AND they get a crash, they might get some actionable data to send back to Netgate, especially if they were recording (for example in PuTTY). My logs didn't show Netgate anything...but serial console output is usually more verbose.

I don't think anyone has actually "bricked" their SG-3100 at this point, during this upgrade, at least no one contributing to this thread.

sdd

@amarand

Not worried about bricking. Just saying I'd love to try it out and report back, but I can only afford a short outage right now. Sounds like there's some risk it might be ~10min to get things going again, and maybe more if for some reason I end up doing a clean install. So I'm taking precautions and will try it out tonight. Doesn't sound like a clean install will be necessary though, and I appreciate your insight.

Amarand

@sdd

For the record, firewall has been rock-solid since the upgrade to the -p1 hotfix.

Had I not been overly cautious, I could have power cycled it in less than a minute after seeing the blinking lights.

NokkieF

@sdd Just make backups before you do anything. You can always import it again.

sdd

@stephenw10

I tried the upgrade tonight, and after more than an hour of working through issues, I went back to 2.4.5-p1.

Here's my experience:

I performed the normal OTA upgrade from the web console.
It rebooted to do the upgrade, and hung here:

mountroot: waiting for device /dev/diskid/DISK-10F88FE8s2a...

This is the "3 lights flashing" issue reported by others in this thread. Since the filesystem hadn't been mounted, I power cycled the device. It mounted on the next boot, and it finished up the update.

After the update, the device wouldn't finish booting. The firewall crashed coming up, and then dropped to a regular login prompt:

Configuring firewall.Segmentation fault (core dumped)
Starting CRON... done.
>>> Removing unnecessary packages... done.
>>> Cleanup pkg cache... done.
Netgate pfSense Plus 21.02-RELEASE (Patch 1) arm Mon Feb 22 09:38:52 EST 2021
Bootup complete

FreeBSD/arm (pfSense.xxx.com) (ttyu0)

login:

I saw this previously with 21.02, but it was intermittent and only happened once or twice. With 21.02-p1 it happens consistently (3/3 tries). I could still get in with ssh, and it gave me the normal text menu.

I did a fresh install from the USB image, and it came up just fine. So then I restored my config.

After doing this, I got one successful boot, and then the firewall started crashing again on every subsequent boot.

When I logged in after the one successful boot, there were about 7 warning notifications telling me about issues with some of my firewall rules. They were rules carried over from 2.4.5-p1, and were related to traffic shaping.

Unfortunately, I don't have time to re-build my firewall and traffic shaping from scratch, so I reverted to 2.4.5-p1 and moved on with my life. I'll check back in a few months once it's stabilized a bit more and try again.

I don't believe the firewall crash is related to the original issue presented here. On a different day I might be able to debug further, collect some logs, and file a separate issue. I work in tech on embedded devices and occasionally on IPv6 routing, but at the end of a long week it's not something I have the energy to do on a Friday night.

On a minor note, I also wanted to call out that upgrading to 21.02-p1 automatically removes the hw.ncpu=1 workaround in /boot/loader.rc.local. I noticed it when I went to remove it. The upgrade overwrites the file with the following:

cat /boot/loader.rc.local

ubenv import
ubenv import boardpn
ubenv import boardrev
ubenv import boardsn
ubenv import eth1addr
ubenv import eth2addr
ubenv import ethaddr

Best regards

stephenw10

Hmm, thank's for documenting that.

The hw.ncpu=1 value should be in /boot/loader.conf.local. That is the file that is specifically carried across an upgrade.

Steve

sdd

@stephenw10

Thanks -- your're right. I'm certain I put the hw.ncpu=1 value in the correct file when I added it, but I clearly looked at the wrong file when I went to remove it. It was removed when I re-installed from the USB recovery image though.

Luketa

Good Morning !!

more reports of update 21.02_1 on SG3100?

I'm still on 2.4.5 p1 and afraid to update.

Amarand

@luketa

Aside from the weird post-install issue a few of us shared here in the thread, the SG-3100 has been rock solid.

If you want to be extra careful, you can use the console cable to monitor the install. If the firewall locks up after installation (“three flashing lights”) check console output to make sure it’s not actually doing something, then, once the console appears idle, power cycle.

Most of the people in this thread went from 2.4.5 p1 to 21.02, experienced the hanging bug from use, added the temporary workaround (limiting the CPU temporarily), which made 21.02 stable enough to hobble along until the hotfix was available.

Now that you can go directly from 2.4.5 to the hotfix, should be fine.

Luketa

@amarand I researched other topics and the staff reported problems with the pfblocker, even in the hotfix version, this SG3100 has active pfblocker and snort.

I think I will wait for more reports.

Luketa

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

Amarand

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

100% stable after install.

I recommend you connect the serial console cable to monitor post-install issues. Some of us here in the thread had “triple flashing lights” hang after install but a power cycle brought everything back up.

mcury

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Amarand

@mcury said in pfSense Plus and SG-3100:

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Were those issues caused by the major jump to 21.02 or the hotfix itself (21.02_1)?

mcury

@amarand said in pfSense Plus and SG-3100:

@mcury said in pfSense Plus and SG-3100:

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Were those issues caused by the major jump to 21.02 or the hotfix itself (21.02_1)?

Netgate worked hard to fix the issues with 21.02, and it is fixed..
This issue with 21.02 was triggered by filter reloads during high load.

The problem with the packages mentioned above was happening with 21.02, and persists on 21.02p1
There are a few redmine bug reports being worked on..

Luketa

@mcury
sad to know that, I use snort and pfblocker, I really won't be able to update, thanks for the information.