21.02-p1 really fix the issue on SG-3100?
-
@solarizde said in 21.02-p1 really fix the issue on SG-3100?:
@bmeeks said in 21.02-p1 really fix the issue on SG-3100?:
messages logged about a Signal 11 crash
Here we go, not so many but still there.
SG-3100, 21.02-RELEASE-p1Packages:
Avahi, Cron, iperf, openvpn-client-export, pfBlockerNG-dev, Service_WatchdogFeb 26 04:31:19 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 26 07:35:22 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Feb 26 07:44:38 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)Thanks! These reports will, I hope, help make the case the problem is really in the PHP engine and not the packages themselves. Snort, Suricata, Unbound and pfBlockerNG-dev are all triggering Signal 11 crashes. And Snort, Suricata and pfBlockerNG-dev are all doing so in the PHP engine. You really should never be able to crash PHP itself.
-
@solarizde Remove pfblockerNG, not sure if only disabling it would solve this issue.
I opened a TAC ticket with Netgate #INC-76936, and they said: "dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:https://redmine.pfsense.org/issues/5413 "
So, they are working on it..
They did a great job with 21.02p1 and I know that they will do it again.. -
@mcury said in 21.02-p1 really fix the issue on SG-3100?:
So, they are working on it..
That redmine Issue is related to a DNS service interrupion which is bad too, but not as bad as the Sig11 on pf.
This is the better place:
redmine.pfsense.org/issues/11444@mcury said in 21.02-p1 really fix the issue on SG-3100?:
Remove pfblockerNG
Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.
-
@solarizde Thanks, I thought that one was already closed, they reopened.
Nice to see that they are taking this seriously :) -
@solarizde said in 21.02-p1 really fix the issue on SG-3100?:
Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.
I'm kind of a noob regarding this technical stuff, cores and such.. but what I understood is that one CPU is trying to read in memory while the other is still writing to it, basically some kind of sync issue between cores, but again, I'm noob and maybe got it all wrong :)
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
PHP bug link
Here is another about the PHP crash/signal 11 on SG-3100.
21.02-p1 fixed a different locking issue in the kernel on SG-3100s.
-
@teamits Perfect. Thank you.
I'm a bit more concerned about some of the other open issues cited here in earlier posts. One bug has been open for 5 years; I hope it's not a dependency.
I was joking the other day with my ex-NSCP coworkers that there's an open Thunderbird (Mozilla) issue in Bugzilla that's been open 20 YEARS and still isn't fixed. https://bugzilla.mozilla.org/show_bug.cgi?id=92165
"[Free software] is only 'free' if your time has no value." - jwz (he was talking about linux)
-
@mcury Basically the 3100 fix was to address a missing "memory barrier" instruction on the arm7 platform.
Since modern CPUs can execute instructions out of order to speed execution, there are times where a process needs to guarantee that all previous instructions are complete (and not being executed still in parallel or out-of-order). This is usually to prevent a race/deadlock condition.
More info here: https://en.wikipedia.org/wiki/Memory_barrier
-
FYI there are two new redmine bugs to track the behavior being seen. Both are related to the FreeBSD php bug.
https://redmine.pfsense.org/issues/11466 "Snort exit with sig 11 on SG-3100"
https://redmine.pfsense.org/issues/11551 "SG-3100 with pfBlockerNG doesn't pass traffic"
This MAY be the tracking bug for the php crash at it was a recent report with FreeBSD 12.1 but the new pfSense 21.02 is using FreeBSD 12.2. The last comment asks if it indeed is a continuing issue on 12.2:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244049
-
Some observations during the Weekend:
hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15h -
@solarizde said in 21.02-p1 really fix the issue on SG-3100?:
Some observations during the Weekend:
hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15hIdentical experience for me on SG-3100, if pfBlocker and two processors enabled then lockup after 6-10hrs. Altering config to 1 cpu has now given me 4 days of stable run time
-
@shadtheman Im also running since Sunday with 2 CPU but pfBlocker disabled, no crash.
-
ok it's defenitly still something wrong with PHP. Yesterday I enabled pfBlocker again, and even running on hw.ncpu = 1 it crashed again:
Mar 6 11:39:21 pfSense syslogd: exiting on signal 15 Mar 6 16:03:29 pfSense kernel: pid 357 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped) Mar 7 04:30:00 pfSense syslogd: exiting on signal 15 Mar 7 04:31:18 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped) Mar 7 09:19:46 pfSense syslogd: exiting on signal 15I will now go to 2 CPUs and disable all packages leaving my pfSense cripled :(
-
I upgraded my SG-3100 to 21.02_1 and pfB-DEVEL _15 this week and I have ZERO php signal 11 messages in my logs. Everything is running smoothly.
You might try upgrading with no snort, suricata and pfB and then re-add them in a default config one by one, then start layering config changes and watching.
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
I upgraded my SG-3100 to 21.02_1 and fsB-DEVEL _15 this week and I have ZERO php signal 11 messages in my logs. Everything is running smoothly.
You might try upgrading with no snort, suricata and pfB and then re-add them in a default config one by one, then start layering config changes and watching.
Did you reboot after installing pfblockerng?
-
@mcury I did but only to check on another bug of unbound not restarting after the update of pfB-devel. I've opened a bug on that issue. Unbound starts properly on boot.
https://redmine.pfsense.org/issues/11632
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
unbound not restarting after the update of pfB-devel
The package maintainer has posted this is a pfSense issue. I can’t find it right now but IIRC it was timing in the package installation. That said it may be occasional as I’ve had it not work a couple times and then one yesterday started fine. The post was in one of the early pfBlocker 3.0.0 version posts I think, or around then. Just check and start after update.
-
@lohphat said in 21.02-p1 really fix the issue on SG-3100?:
@mcury I did but only to check on another bug of unbound not restarting after the update of pfB-devel. I've opened a bug on that issue. Unbound starts properly on boot.
https://redmine.pfsense.org/issues/11632
hm, so pfb 3.0.0_15 is working for you.. Are there other users here that are also running pfblokerng 3.0.0_15 successfully ?
Are you running with default configuration in pfblocker?
-
re: unbound not starting:
https://forum.netgate.com/topic/159094/pfblockerng-v3-0-0_6-update/4
and
https://redmine.pfsense.org/issues/11398
Short answer: check and start it after updating pfBlocker.@mcury said in 21.02-p1 really fix the issue on SG-3100?:
other users here that are also running pfblokerng 3.0.0_15 successfully
We haven't upgraded any SG-3100s but have several in service at our clients so I've been keeping an eye on it. From the various redmine bug reports (at least some linked above) it seems like php-fpm crashes during certain functions (e.g. preg_match) in certain code configurations. My take is it's not a pfBlocker or Snort or Suricata coding issue, it's PHP crashing and that's not going to be very fixable in a package update. Maybe we get lucky and it can be worked around, but it has been a few weeks already. So my personal advice would be for anyone with a 3100 to just be patient and plan to not update for a while, and set System/Update to "previous stable version" if any packages need to be installed or updated, so it doesn't try to install 2.5 packages and dependencies.
-
Upgraded PfBlocker to 2.1.4_25 (just become available) 30 hours ago and have been running happily with both processors enabled for this time, fingers crossed.
https://github.com/pfsense/FreeBSD-ports/commit/b336bf5010920047bf4f607e3b2dfe4d56d9d79f#diff-154b33468fc170ed5c2281d7908ea8f9dc318193eea329feaf5a1df09a4d9da4
