21.02-p1 really fix the issue on SG-3100?

bmeeks

@rsherwood_va said in 21.02-p1 really fix the issue on SG-3100?:

@bmeeks

Here is a screenshot from my dashboard of installed packages. During the upgrade, I removed pfBlockerNG, which is the only one I had installed, and which was disabled in any case.

I do see signal 11 errors in the log:

Feb 17 20:55:30 kernel pid 390 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 17 21:10:38 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 17 21:46:12 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 17 22:50:18 kernel pid 27679 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 17 23:08:17 kernel pid 30028 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 23 10:56:00 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 23 11:11:08 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 23 11:22:52 kernel pid 400 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 02:11:29 kernel pid 54431 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 06:18:04 kernel pid 589 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 06:18:20 kernel pid 32405 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 06:27:51 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 06:33:24 kernel pid 60772 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)

Thank you for the info. That confirms my suspicion.

bmeeks

@lohphat said in 21.02-p1 really fix the issue on SG-3100?:

@bmeeks Do you think this is a php package bug or another kernel lock bug? I don't use snort or suricata but do use pfblockerNG and now that I've seen this issue, I think holding of on 21.02_1 too until it's isolated.

I'm leaning towards it being a PHP bug on 32-bit ARM hardware. Especially since it seems several packages are impacted in similar ways. Whatever this problem is, it's not caused by the latest 21.02_1 update (nor is that update likely to fix it). It looks like something that came in with FreeBSD-12.2/STABLE.

mcury

I'm also affected by it..

Feb 25 19:52:38 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

Removing pfblockerNG-devel completely, solved the problem for me.
I don't use Snort/Suricata

alhh

@mcury

Almost the same here, I was using pfBlockerNG (non-dev).
Tested both: cpu restriction on/off.

Disabling pfBlockerNG (it's still installed) solved the problem.
CPU restriction is NOT necessary.

solarizde

@bmeeks said in 21.02-p1 really fix the issue on SG-3100?:

messages logged about a Signal 11 crash

Here we go, not so many but still there.
SG-3100, 21.02-RELEASE-p1

Packages:
Avahi, Cron, iperf, openvpn-client-export, pfBlockerNG-dev, Service_Watchdog

Feb 26 04:31:19 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:35:22 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:44:38 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

I currently try over night with 21.02-RELEASE-p1 fixed to 1 CPU by loader.conf and pfBlocker enabled. If I enable pfBlocker without the CPU Limit in loader.conf pf stop forwarding traffic within half an hour.

lohphat

@bmeeks Could you please post the FreeBSD PHP bug link and the pfSense tracking bug here for reference so that we can follow?

I think I'm going to hold off until this bug is fixed before upgrading to 21.02_x unless the PHP package can be fixed as part of a package update.

bmeeks

@solarizde said in 21.02-p1 really fix the issue on SG-3100?:

@bmeeks said in 21.02-p1 really fix the issue on SG-3100?:

messages logged about a Signal 11 crash

Here we go, not so many but still there.
SG-3100, 21.02-RELEASE-p1

Packages:
Avahi, Cron, iperf, openvpn-client-export, pfBlockerNG-dev, Service_Watchdog
Feb 26 04:31:19 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:35:22 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:44:38 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

Thanks! These reports will, I hope, help make the case the problem is really in the PHP engine and not the packages themselves. Snort, Suricata, Unbound and pfBlockerNG-dev are all triggering Signal 11 crashes. And Snort, Suricata and pfBlockerNG-dev are all doing so in the PHP engine. You really should never be able to crash PHP itself.

mcury

@solarizde Remove pfblockerNG, not sure if only disabling it would solve this issue.
I opened a TAC ticket with Netgate #INC-76936, and they said: "dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:

https://redmine.pfsense.org/issues/5413 "

So, they are working on it..
They did a great job with 21.02p1 and I know that they will do it again..

solarizde

@mcury said in 21.02-p1 really fix the issue on SG-3100?:

So, they are working on it..

That redmine Issue is related to a DNS service interrupion which is bad too, but not as bad as the Sig11 on pf.

This is the better place:
redmine.pfsense.org/issues/11444

@mcury said in 21.02-p1 really fix the issue on SG-3100?:

Remove pfblockerNG

Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.

mcury

@solarizde Thanks, I thought that one was already closed, they reopened.
Nice to see that they are taking this seriously :)

mcury

@solarizde said in 21.02-p1 really fix the issue on SG-3100?:

Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.

I'm kind of a noob regarding this technical stuff, cores and such.. but what I understood is that one CPU is trying to read in memory while the other is still writing to it, basically some kind of sync issue between cores, but again, I'm noob and maybe got it all wrong :)

SteveITS

@lohphat said in 21.02-p1 really fix the issue on SG-3100?:

PHP bug link

Here is another about the PHP crash/signal 11 on SG-3100.

21.02-p1 fixed a different locking issue in the kernel on SG-3100s.

lohphat

@teamits Perfect. Thank you.

I'm a bit more concerned about some of the other open issues cited here in earlier posts. One bug has been open for 5 years; I hope it's not a dependency.

I was joking the other day with my ex-NSCP coworkers that there's an open Thunderbird (Mozilla) issue in Bugzilla that's been open 20 YEARS and still isn't fixed. https://bugzilla.mozilla.org/show_bug.cgi?id=92165

"[Free software] is only 'free' if your time has no value." - jwz (he was talking about linux)

lohphat

@mcury Basically the 3100 fix was to address a missing "memory barrier" instruction on the arm7 platform.

Since modern CPUs can execute instructions out of order to speed execution, there are times where a process needs to guarantee that all previous instructions are complete (and not being executed still in parallel or out-of-order). This is usually to prevent a race/deadlock condition.

More info here: https://en.wikipedia.org/wiki/Memory_barrier

lohphat

FYI there are two new redmine bugs to track the behavior being seen. Both are related to the FreeBSD php bug.

https://redmine.pfsense.org/issues/11466 "Snort exit with sig 11 on SG-3100"

https://redmine.pfsense.org/issues/11551 "SG-3100 with pfBlockerNG doesn't pass traffic"

This MAY be the tracking bug for the php crash at it was a recent report with FreeBSD 12.1 but the new pfSense 21.02 is using FreeBSD 12.2. The last comment asks if it indeed is a continuing issue on 12.2:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244049

solarizde

Some observations during the Weekend:

hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15h

shadtheman

@solarizde said in 21.02-p1 really fix the issue on SG-3100?:

Some observations during the Weekend:

hw.ncpu=unset, all non default Packages diabled = Stable running 16h without problems
hw.ncpu=unset, pfBlocker-dev and avahi enabled = crash after 1-6h most frequent after pfBlocker update run
hw.ncpu=1, pfBlocker-dev and avahi enabled = stable now since ~15h

Identical experience for me on SG-3100, if pfBlocker and two processors enabled then lockup after 6-10hrs. Altering config to 1 cpu has now given me 4 days of stable run time

solarizde

@shadtheman Im also running since Sunday with 2 CPU but pfBlocker disabled, no crash.

solarizde

ok it's defenitly still something wrong with PHP. Yesterday I enabled pfBlocker again, and even running on hw.ncpu = 1 it crashed again:

Mar  6 11:39:21 pfSense syslogd: exiting on signal 15
Mar  6 16:03:29 pfSense kernel: pid 357 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
Mar  7 04:30:00 pfSense syslogd: exiting on signal 15
Mar  7 04:31:18 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Mar  7 09:19:46 pfSense syslogd: exiting on signal 15

I will now go to 2 CPUs and disable all packages leaving my pfSense cripled :(

lohphat

@solarizde

I upgraded my SG-3100 to 21.02_1 and pfB-DEVEL _15 this week and I have ZERO php signal 11 messages in my logs. Everything is running smoothly.

You might try upgrading with no snort, suricata and pfB and then re-add them in a default config one by one, then start layering config changes and watching.