SG-3100 Crash on Upgrade/Restore when using URL Tables and OpenVPN

artooro · May 19, 2021, 6:55 PM

I have been able to reliably reproduce a crash on the SG-3100 running both pfSense 21.02.2 and on the latest 21.05-RC by importing a specific type of config from a pfSense 2.4.5-p1 box.
This also affects upgrades from 2.4.5 to 21.+

The config needs to have the two following configurations.

A URL table (IPs) alias
An OpenVPN server configured

If both of those are configured (if just one or the other the issue will not occur) then on boot-up you will get a Segmentation fault (core dumped) error and in system.log you will see a line saying something like pfSense kernel: pid 283 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

A couple symptoms will be that the WAN never comes up if it's a static IP and the webGUI never starts so you can't login to webConfigurator.

I'm attaching a demo config here if anyone else wants to try it. This only appears to happen on the SG-3100 from my testing.

Our team has been spending a lot of time fixing failed upgrades to 21.02.2 and I've narrowed down the problems to when the config has both of these items.
And the problems still occur in the 21.05-RC branch.

Example config
pfsense_bug_example.xml

SteveITS · May 19, 2021, 7:56 PM

You're probably finding another trigger for PHP exit with sig 11 on SG-3100, can you post this there?

We've held off upgrading any SG-3100 since we have most of them running IDS and/or pfBlockerNG.

artooro · May 19, 2021, 8:01 PM

@steveits done
We also have held off all upgrades but in some cases the end customer has admin access and decides to upgrade against our direction. And when you maintain hundreds of SG-3100s it happens a few times.

jimp · Jun 28, 2021, 2:16 PM

Can you try the patch to disable PHP PCRE JIT on #11466 Note 32 ?

You can install the System Patches package and then create an entry for the patch URL https://redmine.pfsense.org/attachments/download/3707/patch-disable-pcrejit-arm.diff to apply the fix.

Then run console menu options 16 and 11 to restart PHP and the GUI, or reboot.

Both URL tables and OpenVPN use PCRE matching so both may be fixed by that patch.

artooro · Jun 28, 2021, 2:52 PM

Yep we've been using that patch for several weeks now.

jimp · Jun 28, 2021, 2:54 PM

@artooro said in SG-3100 Crash on Upgrade/Restore when using URL Tables and OpenVPN:

Yep we've been using that patch for several weeks now.

Is everything working OK for you now?

artooro · Jun 28, 2021, 2:54 PM

Yes, we just make sure anything that could cause a problem is disabled before upgrading, and then apply the patch before re-enabling and haven't had the issue recur yet.

jimp · Jun 28, 2021, 2:57 PM

Great!

If you do happen to encounter more PHP crashes, please follow up.

Thanks!