pfSense Plus and SG-3100

Amarand

There was a bug, during the standard upgrade from 2.4.5 to 21.02_2 (the hot fix), where the SG-3100 hung post-install with the triple-flashing-lights.

After connecting a serial console, and verifying no output, I power cycled the SG-3100, and it came back up fine, with the 21.02_2.

I upgraded from 21.02_2 to 21.02.2 (note the _ versus the .) yesterday, and it A) upgraded fine and B) has been running fine with no issues.

Please make sure you take a look at some of the persistent plugin related bugs associated with the 21.x line that are not present in the 2.4.5 code, to make sure you won't be impacted by these bugs, before you upgrade.

Edit: the SG-3100 comes with a serial cable, which you can connect up to a computer and run a serial terminal (like PuTTY or whatever), to watch the upgrade process. I'd recommend doing this, especially from 2.x to 21.x

mcury

@nokkief Can you share more details about your setup?
I've been testing for about 4 days, trying to crash the SG-3100 with high loads, and it seems to be pretty stable..

Are you using suricata/snort ?
Are you using pfblockerng-devel_3.0.0_16, and if so, what feeds?

NokkieF

@mcury
I am seeing this:

2021-04-22 19:30:04.062460+02:00 	kernel 	- 	pid 73327 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 19:15:03.967524+02:00 	kernel 	- 	pid 27459 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 19:00:03.889566+02:00 	kernel 	- 	pid 8141 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 18:45:04.012638+02:00 	kernel 	- 	pid 56654 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 18:30:03.934697+02:00 	kernel 	- 	pid 48931 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 18:15:04.057781+02:00 	kernel 	- 	pid 94150 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 18:02:36.622513+02:00 	php 	25838 	[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
2021-04-22 18:00:04.180824+02:00 	kernel 	- 	pid 25987 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
2021-04-22 18:00:01.142905+02:00 	php 	25838 	[pfBlockerNG] Starting cron process.
2021-04-22 17:45:04.303922+02:00 	kernel 	- 	pid 79696 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) ```
code_text

I am running the following packages
acme security 0.6.9_3
haproxy net 0.61_1
pfBlockerNG-devel net 3.0.0_16

Not sure how I can easily retrieve the lists I am using, but I am using quite a few.

mcury

@nokkief Those are the same errors I was getting before.
I'm suspecting that the number of feeds is causing the crash.

I would like to suggest you to remove pfblockerng-devel completely and reinstall.
There is an option "keep settings" in pfblocker that you shouldn't tick before package removal.

After that, save the pfsense xml configuration, edit it and confirm that there isn't a single entry for pfblockerng...
I'm saying this because the 'old feeds' from previous versions remained in my XML configuration.

After removing all the pfblockerng entries in xml, restore it, then reinstall pfblockerng.

I did add dnsbl_easylists feed after the wizard, for me it is enough.
You should add one feed at a time to test..

NokkieF

Thanks, I will give it a go later. See if I can make them go away. It might also be haproxy, but it might be worth a shot trying

stephenw10

Yes, this looks like it's almost certainly php choking on a large list parsing function for as yet unknown reasons. Loading fewer or smaller lists in pfBlocker will likely avoid it.

NokkieF

@mcury the pfsense config you refer to the one in /conf?

NokkieF

I reinstalled the pfBlocker. I though I cleaned it out pretty well, but it still picked up the PRI1 list (I do not remember this being automatic anyways). So far I do not see any issues in the logging. so it looks good so far. I will give an update later if I see any php errors.

mcury

@nokkief I saved the xml through the GUI settings:

@mcury the pfsense config you refer to the one in /conf?

Save the XML, edit it and search for pfblockerng

You will find a few of these:

			<config>
				<aliasname>ADs_Basic</aliasname>
				<description><![CDATA[ADs Basic - Collection of ADvertisement Domain Feeds.]]></description>
				<row>
					<format>auto</format>
					<state><![CDATA[Enabled]]></state>
					<url>https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts</url>
					<header>StevenBlack_ADs</header>
				</row>
				<action>unbound</action>
				<cron>EveryDay</cron>
				<logging>enabled</logging>
				<order>default</order>
				<dow>1</dow>
			</config>
			<config>
				<aliasname>EasyList</aliasname>
				<description><![CDATA[EasyList Feeds - Utilizing only the domains which are listed to be blocked in full.]]></description>
				<action>unbound</action>
				<cron>EveryDay</cron>
				<dow>1</dow>
				<sort>sort</sort>
				<logging>enabled</logging>
				<order>default</order>
				<filter_alexa></filter_alexa>
				<custom></custom>
				<row>
					<format>auto</format>
					<state><![CDATA[Enabled]]></state>
					<url>https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt</url>
					<header>EasyList</header>
				</row>
				<row>
					<format>auto</format>
					<state><![CDATA[Enabled]]></state>
					<url>https://easylist-downloads.adblockplus.org/easylistportuguese.txt</url>
					<header>EasyList_Portuguese</header>
				</row>
				<row>
					<format>auto</format>
					<state><![CDATA[Enabled]]></state>
					<url>https://easylist.to/easylist/easyprivacy.txt</url>
					<header>EasyPrivacy</header>
				</row>
			</config>
		</pfblockerngdnsbl>

Remove the pfblockernng package without the `keep settings' option.
Remove everything related to pfblockerng from the XML file, and then restore the config to the firewall through the GUI, this will clean all the feeds from the config.

Note: It will take a few minutes to complete, your internet will be down during this period.

Once the config restore/boot completes, install pfblockerng again, run the wizard, and add a few feeds you like, but don't add too many feeds..

I reinstalled the pfBlocker. I though I cleaned it out pretty well, but it still picked up the PRI1 list

PRI1 is OK to have, it's installed during the wizard in 3.0.0_16

I'm using the following lists: (I added Easylist manually, it isn't there by default).

NokkieF

@mcury thanks for the help, i am not seeing the php errors anymore. :)

NokkieF

Can I change the maximum memory consumption for php somewhere? Seems the 3100 has plenty of memory left for some more lists?

NokkieF

I sadly was wrong, just did not pay well enough attention (and probably was hoping it was fixed). I still see the errors every 15 minutes.

stephenw10

Unfortunately this is not directly a memory issue that could be solved like that as I understand it.

Biggy823

@nocling Is Snort fixed in this version?

mcury

@nokkief said in pfSense Plus and SG-3100:

I sadly was wrong, just did not pay well enough attention (and probably was hoping it was fixed). I still see the errors every 15 minutes.

Strange, I'm not getting those here.. More than a week running pfblocker, in several vlans..

15 minutes? Do you have a cron set to run every 15 minutes?

bmeeks

@biggy823 said in pfSense Plus and SG-3100:

@nocling Is Snort fixed in this version?

No. Snort and Suricata both are still victims of the PHP bug on ARM 32-bit hardware such as the SG-3100.

NokkieF

@mcury Not that I am aware of. Where can I see the cronjobs listed for the pfSense? crontab -l shows no cronjobs for root
cat of /etc/crontab shows
*/1 * * * * root /usr/sbin/newsyslog

mcury

@nokkief said in pfSense Plus and SG-3100:

@mcury Not that I am aware of. Where can I see the cronjobs listed for the pfSense? crontab -l shows no cronjobs for root
cat of /etc/crontab shows
*/1 * * * * root /usr/sbin/newsyslog

There is a package in package manager, cron
Install cron and check what is running every 15 minutes.. It will show all cron tasks
Did you set pfblockerng to update the lists every 15 minutes?

NokkieF

@mcury No, once a day. I will instal cron and see what it does

NokkieF

Looks like it's doing this
0,15,30,45 * * * * root /etc/rc.filter_configure_sync