21.02-p1 really fix the issue on SG-3100?
-
@mcury Basically the 3100 fix was to address a missing "memory barrier" instruction on the arm7 platform.
Since modern CPUs can execute instructions out of order to speed execution, there are times where a process needs to guarantee that all previous instructions are complete (and not being executed still in parallel or out-of-order). This is usually to prevent a race/deadlock condition.
More info here: https://en.wikipedia.org/wiki/Memory_barrier
-
FYI there are two new redmine bugs to track the behavior being seen. Both are related to the FreeBSD php bug.
https://redmine.pfsense.org/issues/11466 "Snort exit with sig 11 on SG-3100"
https://redmine.pfsense.org/issues/11551 "SG-3100 with pfBlockerNG doesn't pass traffic"
This MAY be the tracking bug for the php crash at it was a recent report with FreeBSD 12.1 but the new pfSense 21.02 is using FreeBSD 12.2. The last comment asks if it indeed is a continuing issue on 12.2:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244049
-
Some observations during the Weekend:
hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15h -
@solarizde said in 21.02-p1 really fix the issue on SG-3100?:
Some observations during the Weekend:
hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15hIdentical experience for me on SG-3100, if pfBlocker and two processors enabled then lockup after 6-10hrs. Altering config to 1 cpu has now given me 4 days of stable run time
-
@shadtheman Im also running since Sunday with 2 CPU but pfBlocker disabled, no crash.
-
ok it's defenitly still something wrong with PHP. Yesterday I enabled pfBlocker again, and even running on hw.ncpu = 1 it crashed again:
Mar 6 11:39:21 pfSense syslogd: exiting on signal 15 Mar 6 16:03:29 pfSense kernel: pid 357 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) Mar 7 04:30:00 pfSense syslogd: exiting on signal 15 Mar 7 04:31:18 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Mar 7 09:19:46 pfSense syslogd: exiting on signal 15I will now go to 2 CPUs and disable all packages leaving my pfSense cripled :(
-
I upgraded my SG-3100 to 21.02_1 and pfB-DEVEL _15 this week and I have ZERO php signal 11 messages in my logs. Everything is running smoothly.
You might try upgrading with no snort, suricata and pfB and then re-add them in a default config one by one, then start layering config changes and watching.
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
I upgraded my SG-3100 to 21.02_1 and fsB-DEVEL _15 this week and I have ZERO php signal 11 messages in my logs. Everything is running smoothly.
You might try upgrading with no snort, suricata and pfB and then re-add them in a default config one by one, then start layering config changes and watching.
Did you reboot after installing pfblockerng?
-
@mcury I did but only to check on another bug of unbound not restarting after the update of pfB-devel. I've opened a bug on that issue. Unbound starts properly on boot.
https://redmine.pfsense.org/issues/11632
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
unbound not restarting after the update of pfB-devel
The package maintainer has posted this is a pfSense issue. I can’t find it right now but IIRC it was timing in the package installation. That said it may be occasional as I’ve had it not work a couple times and then one yesterday started fine. The post was in one of the early pfBlocker 3.0.0 version posts I think, or around then. Just check and start after update.
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
@mcury I did but only to check on another bug of unbound not restarting after the update of pfB-devel. I've opened a bug on that issue. Unbound starts properly on boot.
https://redmine.pfsense.org/issues/11632
hm, so pfb 3.0.0_15 is working for you.. Are there other users here that are also running pfblokerng 3.0.0_15 successfully ?
Are you running with default configuration in pfblocker?
-
re: unbound not starting:
https://forum.netgate.com/topic/159094/pfblockerng-v3-0-0_6-update/4
and
https://redmine.pfsense.org/issues/11398
Short answer: check and start it after updating pfBlocker.@mcury said in 21.02-p1 really fix the issue on SG-3100?:
other users here that are also running pfblokerng 3.0.0_15 successfully
We haven't upgraded any SG-3100s but have several in service at our clients so I've been keeping an eye on it. From the various redmine bug reports (at least some linked above) it seems like php-fpm crashes during certain functions (e.g. preg_match) in certain code configurations. My take is it's not a pfBlocker or Snort or Suricata coding issue, it's PHP crashing and that's not going to be very fixable in a package update. Maybe we get lucky and it can be worked around, but it has been a few weeks already. So my personal advice would be for anyone with a 3100 to just be patient and plan to not update for a while, and set System/Update to "previous stable version" if any packages need to be installed or updated, so it doesn't try to install 2.5 packages and dependencies.
-
Upgraded PfBlocker to 2.1.4_25 (just become available) 30 hours ago and have been running happily with both processors enabled for this time, fingers crossed.
https://github.com/pfsense/FreeBSD-ports/commit/b336bf5010920047bf4f607e3b2dfe4d56d9d79f#diff-154b33468fc170ed5c2281d7908ea8f9dc318193eea329feaf5a1df09a4d9da4
-
Hi,
I have upgraded to the 21.02-RELEASE-p1 (arm) a while ago. I tried installing surricata, snort, zeek, they all crash after a while. I do also see the bug with the php. Every 15 minutes like clockwork it seems to crash. I do not get lock ups on the firewall, it seems to run mostly fine other than the crashed of php.
Is there any known issue for this? Or am I replying on an old thread?
-
To clarify the above, I am no longer running snort, surricata or zeek. I am running pfBlocker however.
I have also noticed the php feeling rather sluggish, but that could just be me being impatient. -
@nokkief said in 21.02-p1 really fix the issue on SG-3100?:
Is there any known issue for this? Or am I replying on an old thread?
This is the thread about that. See the Redmine links to the bug reports above, e.g.:
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
https://redmine.pfsense.org/issues/11466 "Snort exit with sig 11 on SG-3100"
Basically PHP on the 3100 has some issues with certain functions that crashes PHP. re: every 15 minutes, do you have something scheduled for every 15 minutes?
-
@teamits Yeah, but the #11444 is about it halting the firewall. I had that before, but not after the latest patch. This one (issue 11444) also has been marked as resolved.
The issue however remains for the php crashing for me. I do not recall anything being scheduled every 15 minutes. pfBlocker updates at intervals of at least 4 hours or more. it says it is a kernel dump. Could it be wireguard? I just realized it is running, although I have no clients connected to it right now.
-
@teamits
Oh, I am blind. I now see the other issue mentioned, apologies
