Netgate 3100 + PHP Crashes

jimp

A few weeks ago I posted a patch to disable PHP PCRE JIT on #11466 Note 32 which works around most, if not all, PHP crashes on Netgate 3100 devices.

We have had some feedback on it, mostly positive, but a few people still have issues but haven't followed up with details about what still isn't working.

Affected users can install the System Patches package and then create an entry for the patch URL https://redmine.pfsense.org/attachments/download/3707/patch-disable-pcrejit-arm.diff to apply the fix.

To ensure everything is restarted properly with the patch active, reboot the device after applying the patch.

Several parts of the base system and packages use PCRE matching so it's possible to encounter the problem in a number of ways, such as from using OpenVPN, URL Table aliases, Snort, Suricata, pfBlockerNG, and more.

Most of these should be fixed by the patch but we are still seeing some reports of issues and could use additional feedback.

Snort: PHP crashes are fixed by the patch, but there are other issues such as snort itself crashing with signal 10 which are still present on the 3100 -- We don't need further details here, as that is being tracked in its own thread already.
Suricata: PHP crashes are fixed by the patch, but there are still a couple reports that suricata is not working properly, or exits after running for a while. We need more information on this as there have been no follow-up messages with details about current issues, no log messages, etc.
pfBlockerNG: Multiple reports that all issues are fixed by the patch and things are operating correctly. A small number of users report ongoing issues, but again, no details, no log messages, etc. We need more information about problems users are still encountering here.
OpenVPN: PHP crashes are fixed by the patch, no reports of ongoing issues thus far.
URL Table Aliases: PHP crashes are fixed by the patch, no reports of ongoing issues thus far.

Anyone who still has issues with these or other areas, please let us know with as much accompanying detail as possible what is still not working along with any error messages, log messages, or other details. For packages, describe at least the specific version (e.g. regular or -devel, plus version numbers), which features you are using, and if it was known to work for you in past versions with an identical configuration.

Before reporting problems, please ensure the patch applied correctly, and that the device was rebooted after applying the patch.

Thanks!

johnhoddinott

Hopefully helpful:
I'm having the issue too, netgate 3100 php crashes with errors like this every time:

Jul 16 23:04:29 kernel pid 2131 (php), jid 0, uid 0: exited on signal 11 (core dumped)
Jul 16 23:04:42 php-fpm 62389 Starting Suricata on WAN(pppoe0) per user request...
Jul 16 23:04:42 php 13115 [Suricata] Updating rules configuration for: WAN ...
Jul 16 23:04:46 kernel pid 13115 (php), jid 0, uid 0: exited on signal 11 (core dumped)

I'm running the latest version of pfsense with only suricata as the additional package and only the basic ET rules ('Install ETOpen Emerging Threats rules' is enabled)
I'm also using my OPT1 interface interface as a SPAN port and i've made a few minor tweaks to DNS Resolver and some static DHCP reservations

Last night I enabled suricata for the first time, found the patch referenced above, applied it rebooted
suricata was running on the interface. great.
I disabled suricata on the interface

this evening I tried to re-enable suricata on the interface and the error was back.
I reverted the patch, re applied it, I set the patch to auto apply, rebooted and again suricata was running on the interface again.

for some added troubleshooting information:
I added another interface by cloning the one referenced above, starting suricata on that inferface worked fine
also stopping and starting suricata on the interface referenced above worked fine also

Again, I hope this feedback helps other users.
it's great that the patch was available and not hard to find after a quick search for the error code.

-John

stephenw10

You should only need to apply the patch once, it should survive a reboot. The auto-apply option only applies to a firmware update when a patch is likely to be overwritten.
But all data is good! Thanks for reporting.

Steve

Luketa

This post is deleted!

Rich Taylor-Worth

SG-3100 does not crash with suricata as often. I would say that now it is not related to PHP but to the complexity of my local network causing memory overflow problems. I use one isolation vlan for testing client's machines; One IOT vlan for all that stuff and one vlan for my WEEWX weather app on a raspberry pi. I then use a main lan for computers. All are running on my poor little SG-3100.

I did give up on SNORT.

I never could make that run and I only run Suricata interfaces on the WAN and LAN interfaces which seems now to be stable. Suricata did exit after a while when I ran it on multiple vlans as memory crept up even when I manually added a large swap drive that eventually filled and Suricata died. Now that I run it on just two interfaces I do not have that problem and have even gotten rid of the swap drive.

Why spend money until I have to upgrade hardware? As I read the specs I would have to jump up a couple of levels to get the same throughput and here in Chattanooga TN USA my GIG to the internet is cheap.

Rich

Luketa

@rich-taylor-worth
I installed meerkat, couldn't use snort supress, I'm having to evaluate everything again. Too bad snort didn't run on 21.05 , it was working perfectly on 2.4.5-p1

Rich Taylor-Worth

@luketa
Agreed-
but this is the "joy" of modern code development and use. Sure at least we can vet a lot of the code ourselves or have people we trust that vet it for us but even in the closed source world multiple vendors support multiple pieces and who the hell knows who maintains that library that you use in a critical way.

I have seen a cartoon where this monstrous pile of different colored boxes labeled code libraries was supported by a stick labeled "open source plugin maintained for free by Frank." :-)

MaxK 0

@jimp I see that 11466 has been closed with the following comments:

"Per Mateusz, PHP JIT will need to be disabled on the 3100. There is currently no other way around the crash on multi-CPU 32-bit ARM systems."

"Since disabling JIT is the best solution in this case, this issue can now be closed."

Does this mean PHP JIT will be permanently disabled on the 3100 going forward?

jimp

Yes