Strange error: There were error(s) loading the rules: pfctl: pfctl_rules

[kprovost]

@bmeeks That could potentially also be causing a ticket mismatch in DIOCADDRULENV. There's a ticket number check on the ruleset, and another one for the address list. The pfctl -x loud enabled kernel output should let us figure out which one.

Given the presence of the DIOCRADDADDRS call from snort I'm 95% confident it'll be the pool ticket, but it'd be good to confirm anyway.

[kprovost]

If people are feeling adventurous there's an experimental (amd64) kernel here: http://pfsense-build-01.netgate.com/~kp/kernel.tar.bz2

It should prevent pf from getting stuck, where it's unable to update rules.

Extract that in /boot/. It will overwrite the existing kernel there, so ideally test this on test machines, or snapshotted VM's.
(To be clear: this is an amd64 kernel, do not try it on 3100 or 1100 or 2100, because it will not work there.)

[Gertjan]

@kprovost said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:

feeling adventurous

Always.

@kprovost said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:

http://pfsense-build-01.netgate.com/~kp/kernel.tar.bz2

http ? Thought that was ditched some time ago.

pfsense-build-01.netgate.com
is not accessible for the common morsels.

[kprovost]

@gertjan Sorry, I thought that one was public.

Let's try https://people.freebsd.org/~kp/kernel.tar.bz2 then.

[artooro]

@kprovost I did get one today with pfctl -x loud executed first. Attached here in case it's still helpful.

1-truss_pfctl_1660896573.txt

I see the kernel you uploaded, I'll install it on the slave node in the HA cluster, it doesn't happen as often on the slave but at least it's not as risky to test it.

[Flole]

I believe the "real" errors can be seen in dmesg now, you should at least save the dmesg output somewhere in case it's needed.

[artooro]

@flole in dmesg right now all I'm seeing is pf: wire key attach failed on all messages. Not sure whether it's related at all.
If it's still helpful I can write some more code to capture it at the time of incident.

[djrobx]

This is happening to me too on 22.05. Same "busy" message:

/root: pfctl -Fa
pfctl: pfctl_clear_eth_rules: Device busy

Mine popped up when trying to modify OpenPVN client settings.

Mine's as close to a virgin install as you can get on self-supplied hardware (2.6.0->22.01->22.05). It ran most of the day on 22.01 with no problem, then I upgraded to latest.

No custom packages, have not touched the file system other than to load one script back in.

[artooro]

@kprovost is it possible to get the kernel patch for armv7 (for the SG-3100) as most installs I have exhibiting the issue are using that platform.

[kprovost]

@artooro Here's a kernel for the 3100. https://people.freebsd.org/~kp/kernel-3100.tar.bz2

I have NOT tested this kernel as I don't have a 3100. Be careful to ensure you don't break your device.

[artooro]

@kprovost after installing this kernel patch I was able to observe a collision of pf syscalls and it did not end up in a locked state like it did previously.
So far I'd say this patch is doing the job.

[djrobx]

@kprovost I have also been running with the kernel patch. It seems to have resolved the problem for me as well.

[Flole]

Is this intended as "proper" fix or just as a temporary workaround? Or asked differently: Will this be merged like this or will there be a different fix? Is there a diff available somewhere so I can see what was changed?

[stephenw10]

Right now these are test kernels just prove we have found the issue. Now that appears to be the case we will merge it and look at what we can do in existing 22.05 installs.

Steve

[kprovost]

@flole It's a real fix, not a workaround. It's gone in upstream: https://cgit.freebsd.org/src/commit/?id=6ab80e7275091c900da8d2e84a7b0bb4c34a1e41

and I'll merge it to our local branch just as soon as this test-build finishes.

[artooro]

@kprovost would it be possible to also get the kernel patch for ARM64 as I have Netgate 2100s and a 1100 that also have this happening.
Thanks for all your help!

[kprovost]

@artooro We don't need any further testing on different platforms.
The fix has been merged in all relevant branches (and upstream FreeBSD) and will be present in upcoming snapshots, when they're published again.

[bblacey]

@kprovost

Apologies for bumping this relatively old thread but I'm seeing this on a new Netgate 6100 Max running pfSense+ 22.05-RELEASE. Is there a snapshot available that effectively has only this one merge included? This is a production machine so I want to keep the non-release deltas to a minimum.

[kprovost]

@bblacey I don't believe so, no.

[mardacs27]

@stephenw10

Any update or tutorial on this? Constantly happening on my SG-2440