pfSense Plus and SG-3100

sdd

I tried the upgrade tonight, and after more than an hour of working through issues, I went back to 2.4.5-p1.

Here's my experience:

I performed the normal OTA upgrade from the web console.
It rebooted to do the upgrade, and hung here:

mountroot: waiting for device /dev/diskid/DISK-10F88FE8s2a...

This is the "3 lights flashing" issue reported by others in this thread. Since the filesystem hadn't been mounted, I power cycled the device. It mounted on the next boot, and it finished up the update.

After the update, the device wouldn't finish booting. The firewall crashed coming up, and then dropped to a regular login prompt:

Configuring firewall.Segmentation fault (core dumped)
Starting CRON... done.
>>> Removing unnecessary packages... done.
>>> Cleanup pkg cache... done.
Netgate pfSense Plus 21.02-RELEASE (Patch 1) arm Mon Feb 22 09:38:52 EST 2021
Bootup complete

FreeBSD/arm (pfSense.xxx.com) (ttyu0)

login:

I saw this previously with 21.02, but it was intermittent and only happened once or twice. With 21.02-p1 it happens consistently (3/3 tries). I could still get in with ssh, and it gave me the normal text menu.

I did a fresh install from the USB image, and it came up just fine. So then I restored my config.

After doing this, I got one successful boot, and then the firewall started crashing again on every subsequent boot.

When I logged in after the one successful boot, there were about 7 warning notifications telling me about issues with some of my firewall rules. They were rules carried over from 2.4.5-p1, and were related to traffic shaping.

Unfortunately, I don't have time to re-build my firewall and traffic shaping from scratch, so I reverted to 2.4.5-p1 and moved on with my life. I'll check back in a few months once it's stabilized a bit more and try again.

I don't believe the firewall crash is related to the original issue presented here. On a different day I might be able to debug further, collect some logs, and file a separate issue. I work in tech on embedded devices and occasionally on IPv6 routing, but at the end of a long week it's not something I have the energy to do on a Friday night.

On a minor note, I also wanted to call out that upgrading to 21.02-p1 automatically removes the hw.ncpu=1 workaround in /boot/loader.rc.local. I noticed it when I went to remove it. The upgrade overwrites the file with the following:

cat /boot/loader.rc.local

ubenv import
ubenv import boardpn
ubenv import boardrev
ubenv import boardsn
ubenv import eth1addr
ubenv import eth2addr
ubenv import ethaddr

Best regards

stephenw10

Hmm, thank's for documenting that.

The hw.ncpu=1 value should be in /boot/loader.conf.local. That is the file that is specifically carried across an upgrade.

Steve

sdd

@stephenw10

Thanks -- your're right. I'm certain I put the hw.ncpu=1 value in the correct file when I added it, but I clearly looked at the wrong file when I went to remove it. It was removed when I re-installed from the USB recovery image though.

Luketa

Good Morning !!

more reports of update 21.02_1 on SG3100?

I'm still on 2.4.5 p1 and afraid to update.

Amarand

@luketa

Aside from the weird post-install issue a few of us shared here in the thread, the SG-3100 has been rock solid.

If you want to be extra careful, you can use the console cable to monitor the install. If the firewall locks up after installation (“three flashing lights”) check console output to make sure it’s not actually doing something, then, once the console appears idle, power cycle.

Most of the people in this thread went from 2.4.5 p1 to 21.02, experienced the hanging bug from use, added the temporary workaround (limiting the CPU temporarily), which made 21.02 stable enough to hobble along until the hotfix was available.

Now that you can go directly from 2.4.5 to the hotfix, should be fine.

Luketa

@amarand I researched other topics and the staff reported problems with the pfblocker, even in the hotfix version, this SG3100 has active pfblocker and snort.

I think I will wait for more reports.

Luketa

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

Amarand

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

100% stable after install.

I recommend you connect the serial console cable to monitor post-install issues. Some of us here in the thread had “triple flashing lights” hang after install but a power cycle brought everything back up.

mcury

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Amarand

@mcury said in pfSense Plus and SG-3100:

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Were those issues caused by the major jump to 21.02 or the hotfix itself (21.02_1)?

mcury

@amarand said in pfSense Plus and SG-3100:

@mcury said in pfSense Plus and SG-3100:

@luketa said in pfSense Plus and SG-3100:

Hey guys, how is the SG3100 on v 21.02_1, is it stable?

I intend to update this weekend.

It's stable..
There are problems with packages like pfblockerng, snot/suricata, so I wouldn't recommend you to upgrade in case you use these packages.
Also, there are some problems with IPsec, and a few patches were released to address these issues.

Patches that I found so far that can help you with IPsec/Openvpn
https://forum.netgate.com/post/971136

Were those issues caused by the major jump to 21.02 or the hotfix itself (21.02_1)?

Netgate worked hard to fix the issues with 21.02, and it is fixed..
This issue with 21.02 was triggered by filter reloads during high load.

The problem with the packages mentioned above was happening with 21.02, and persists on 21.02p1
There are a few redmine bug reports being worked on..

Luketa

@mcury
sad to know that, I use snort and pfblocker, I really won't be able to update, thanks for the information.

CyberDude6

Not sure if I did something wrong but my SG-3100 just hangs there with the three blinking lights. I thought this was fixed but I guess not. Console cable connection on putty shows just a bunch of dots, pretty much constantly. Any ideas?

Amarand

@cyberdude6 said in pfSense Plus and SG-3100:

Not sure if I did something wrong but my SG-3100 just hangs there with the three blinking lights. I thought this was fixed but I guess not. Console cable connection on putty shows just a bunch of dots, pretty much constantly. Any ideas?

If the dots are still outputting, wait until they cease. That might be an indication that it’s still writing the latest update?

Once there’s no output, if you still have three blinking lights, you can power cycle.

I would not power cycle if there were any indication of activity on the console.

stephenw10

Did you have the console connected up before it rebooted? Any idea what the last it showed was before dots?

Continuous single dots like that is not part of any update I would expect. Sounds like it's waiting for something maybe.

Steve

CyberDude6

@stephenw10

I ended up just leaving the SG3100 there and came back a few hours later. It ended up stopping that nonsense and went to the normal menu to do all the usual things. Was able to factory restore there and it is working well now!

amarcino

I use both PF Blocker and Suricata, so I am going to wait on this update.

Luketa

@amarcino I'm in the same situation.
because of things, I think I will stay for a long time in 2.4.5

stephenw10

The PHP issue, which is what you will be hitting with pfBlocker and Suricata, seems to be very load dependent.
I have both those running here on SG-3100s in 21.02p1 without any problems. The throughput on the those is not huge and I don't have a large number of lists loaded though.

Steve

mcury

@stephenw10 said in pfSense Plus and SG-3100:

The PHP issue, which is what you will be hitting with pfBlocker and Suricata, seems to be very load dependent.
I have both those running here on SG-3100s in 21.02p1 without any problems. The throughput on the those is not huge and I don't have a large number of lists loaded though.

Steve

Hello Steve, if you install pfblockerng, run the wizard, and reboot, you get:

done.
Configuring firewall.
pflog0: promiscuous mode enabled
pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Segmentation fault (core dumped)
Starting CRON... done.
 Starting package IPsec Profile Wizard...done.
 Starting package AWS VPC Wizard...done.
 Starting package acme...done.

Nothing works, to recover access, you need, through the console, select the option 11, disable pfblockerng and reboot again.