pfSense Plus and SG-3100

mcury

@nokkief said in pfSense Plus and SG-3100:

@mcury Yeah, I am
21.02.2-RELEASE (arm)
built on Mon Apr 12 07:50:07 EDT 2021
FreeBSD 12.2-STABLE

The system is on the latest version.
Version information updated at Mon Apr 26 9:02:50 CEST 2021

hm, it is strange..
now I'm not sure if there is something particular about your setup that is triggering it, or if I'm not triggering it by luck and this bug is waiting to happen here, hidden..

Edit: Did you enable these options in DNS Resolver?

NokkieF

@mcury No, but I did enable the python module. But with or without is not making a difference.

s0m3f00l

@bmeeks said in pfSense Plus and SG-3100:

PHP bug on ARM 32-bit

Has anyone tried rebuilding php from source for the 3100? I am still stuck on 2.4.5p1 because we use suricata. If I wanted just a router firewall without any NGFW stuff I could have build an openwrt box.

artooro

@stephenw10 we've had issues with several SG-3100s upgrading them to pfSense 21.02.2, just working on one right now. I just tried clearing all the files in /var/db/aliastables to ensure that it wasn't a large table causing the issue but after a reboot I am still seeing kernel: pid 404 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

The two symptoms I've seen after multiple SG-3100 upgrades is that

The default gateway is missing
When connecting via console the menu interface doesn't load and it jumps straight to a shell.

Am I correct in there being no workaround at this time for these SG-3100 issues?

artooro

I just found that commenting out the following lines from /etc/rc.bootup resolved the php-cgi crash issue and on reboot both the default route and console are working correctly.
I figured it must be OpenVPN because of when the error shows up in system.log.

//echo "Syncing OpenVPN settings...";
//openvpn_resync_all();
//echo "done.\n";

Still need to determine what about OpenVPN would be triggering this.

NokkieF

@artooro Thanks, I will give it a try and see if it 'fixes' the error for me too.

artooro

Doesn't look like it helped anything. After another reboot php-cgi started to core dump when simply syncing the firewall. Now getting Configuring firewall.Segmentation fault (core dumped)

So far we've had to re-install the OS on quite a few SG-3100 appliances and currently have a policy of keeping them on 2.4.5-p1 until there is a more stable pfSense release.

NokkieF

@artooro Sadly I had the same result, the firewall was worse off than before. I decided to just roll back the whole thing to factory settings. I am now running pfBlocker with a bunch of lists without any issues. I no longer see the php errors.
I will add haproxy and the openvpn later to see if it screws it up again.

Luketa

@artooro @NokkieF ,
I am researching a lot before updating 2.4.5-p1, and I came to the conclusion that I am not going to update, my client is a 24 hour clinic and with this many problems I can get complicated.

NokkieF

So after I went back to the factory settings and set it up again I had no problems with pfBlocker installed. Recently I set up an OpenVPN client and I am seeing the php cor edumps again. So maybe it is related to openvpn?

bmeeks

The problem is within PHP itself on 32-bit ARM platforms. It will manifest itself in different packages at somewhat randomly differing places. Almost every installed package in pfSense makes use of PHP GUI code at some point. Depending on exactly what parts and pieces of PHP are in operation at any given moment, the bug may surface- or it may not.

In the case of the Snort and Suricata packages, the bug is triggered by successive calls to the preg_match() regex function of PHP.

So trying to find a particular package to blame is pointless, unless you pin it where the actual problem lives -- in PHP itself. Adding or removing particular packages, or reinstalling them, or reconfiguring them may appear to give temporary relief, but it's not a real fix. The only fix is within PHP itself.

If you remove all packages and run a plain-vanilla pfSense setup on your SG-3100, it will probably run fine. That's because the base pfSense code, for the most part, appears to not be using the PHP functions that cause the problem on 32-bit ARM platforms. But a number of the add-on packages will call those troublesome PHP functions and thus trigger the bug.

artooro

Looks like pfSense Plus 21.05 just came out, and this issue is still not fixed. And no word from anyone at Netgate either in this forum or the redmine issue.

jimp

Try the patch to disable PHP PCRE JIT on #11466 Note 32.

You can install the System Patches package and then create an entry for the patch URL https://redmine.pfsense.org/attachments/download/3707/patch-disable-pcrejit-arm.diff to apply the fix.

Then run console menu options 16 and 11 to restart PHP and the GUI, or reboot.

mcury

Applied the patch, thanks Jimp.

Can you confirm if the D28821 fix was ported to 21.05 version?
I got a few php dumps before installing packets, while reloading filters.

jimp

@mcury said in pfSense Plus and SG-3100:

Applied the patch, thanks Jimp.

Can you confirm if the D28821 fix was ported to 21.05 version?
I got a few php dumps before installing packets, while reloading filters.

Yes, the patch from https://reviews.freebsd.org/D28821 is present in the FreeBSD source for 21.05.

Cornel

@jimp has this patch been confirmed as a workaround for the php issue?

NokkieF

@jimp Do I need to run the patch for the new release, or is it already part of it?

SteveITS

@cornel The patch for the PHP crashing issue is in note 32 here and is not in 21.05. Apply via the System Patches package.

mcury

I was getting php errors in 21.05, so I applied this patch.. No more problems noticed.. It seems pretty stable now..
pfblockerng is running perfectly with this patch, along with openvpn and ipsec.

Didn't test snort/suricata..

Cornel

@mcury Thanks!

I bit the bullet and pressed update...

Update was a total failure. Fortunately the pfSense support team is very responsive. A link for the download image was sent in the same minute as I created the ticket. Installed image, restored from backup, applied the patch and up and running again!

Errors I got in console after I pressed update were many lines of:
UFS /dev/diskid/DISK-E937110023s2a (/) cylinder checksum failed: cg 124, cgp: 0x0 != bp: 0xe911d0ad
UFS /dev/diskid/DISK-E937110023s2a (/) cylinder checksum failed: cg 125, cgp: 0x0 != bp: 0x381abb22
UFS /dev/diskid/DISK-E937110023s2a (/) cylinder checksum failed: cg 126, cgp: 0x0 != bp: 0x4eeb7142