Strange error: There were error(s) loading the rules: pfctl: pfctl_rules
-
@stephenw10 I've ruled out a mismatch (and it worked after the update, so this is most likely not related to the update). The system was rebooted and then this apparently appeared.
The extra -v flags do cause that os fingerprint message to appear, but then only the pfctl_rules thing again with an exit code of 1.
-
Can you run any pfctl command? Like:
pfctl -sr -
@stephenw10 -sr is quiet but exit code is 0. When I do -sa I even get my configured aliases as a table, so apparently it all worked for a short time after rebooting so the rules and aliases were loaded properly.
-
This is in 2.6 on custom hardware?
-
@stephenw10 No, it's on 22.05. I'm sure a downgrade would fix it and I have the recovery image here already, but it would probably be better to do further debugging to see what causes pfctl to get in a state where it not longer wants to accept rules.
-
The only time I've been able to explain that is when there was a kernell mismatch causing pfctl not to function as expected with pf in the kernel.
Try:
[22.05-RELEASE][admin@4100-2.stevew.lan]/root: sha256 /sbin/pfctl SHA256 (/sbin/pfctl) = 4f9310145dfe739126392d77e7cb37d8cf845317f10624fb8b2fd1e408323761 [22.05-RELEASE][admin@4100-2.stevew.lan]/root: uname -a FreeBSD 4100-2.stevew.lan 12.3-STABLE FreeBSD 12.3-STABLE plus-RELENG_22_05-n202700-3ddaea61055 pfSense amd64 [22.05-RELEASE][admin@4100-2.stevew.lan]/root: freebsd-version -kur 12.3-STABLE 12.3-STABLE 12.3-STABLE -
That matches:
[22.05-RELEASE][root@XXXXXXX]/root: sha256 /sbin/pfctl SHA256 (/sbin/pfctl) = 4f9310145dfe739126392d77e7cb37d8cf845317f10624fb8b2fd1e408323761 [22.05-RELEASE][root@XXXXXXX]/root: uname -a FreeBSD XXXXXXXXXXXX 12.3-STABLE FreeBSD 12.3-STABLE plus-RELENG_22_05-n202700-3ddaea61055 pfSense amd64 [22.05-RELEASE][root@XXXXXXX]/root: freebsd-version -kur 12.3-STABLE 12.3-STABLE 12.3-STABLE -
Hmm, that's on custom hardware? Updated from CE to Plus?
-
It was on Plus 22.01, then it got upgraded to 22.05 and it worked perfectly fine. Then there was a power outage so the firewall got shutdown (graceful shutdown when the UPS ran out of battery) and after it booted again this issue started to appear. As there are multiple users affected by this apparently and rebooting sometimes seems to solve it (at least for others, for me it doesn't for some reason) there is probably some nasty bug somewhere. Shouldn't pfctl normally return an error message if something is not right?
-
It does if there's an error in the ruleset, yes. It doesn't if the rules file is empty though:
[22.05-RELEASE][admin@4100-2.stevew.lan]/root: pfctl -vvvf /tmp/rules.bad Loaded 762 passive OS fingerprints [22.05-RELEASE][admin@4100-2.stevew.lan]/root: echo $? 0We have seen devices load a kernel from another storage device but I've never been able to replicate that here. Obviously that only applies if you have more than one drive in the system.
And it doesn't appear to have happened here since the ported kernel is correct.However the other symptoms point to that.
Are there any errors in the boot log?
-
@stephenw10 said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
It does if there's an error in the ruleset, yes. It doesn't if the rules file is empty though:
You are getting exit code 0 there though, I am getting exit code 1 there.
@stephenw10 said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
We have seen devices load a kernel from another storage device but I've never been able to replicate that here. Obviously that only applies if you have more than one drive in the system.
It's a single drive only. I always wanted to migrate to ZFS though and use mirroring on this system, so maybe it's a good time now to backup the config and reinstall using ZFS on 2 drives.
I just saw in the system log this message from snort being logged:
s2c_pf_block() => ioctl() DIOCRADDADDRS: No such processSo maybe this gives more information on what's going on? I assume the same ioctl call is being made by pfctl aswell and most likely the same error is returned there? Just that pfctl isn't displaying that error.
In the boot log I am only seeing this error, but it seems unrelated:
dummynet: bad switch 21! -
Yeah those errors all point to a mismatch in something but I'm unsure what if the kernel and pfctl are correct.
The quickest way back up is going to be a re-install though.
Steve
-
Maybe it's the library thats not matching? Snort doesn't call pfctl I think, if it uses the library which pfctl also uses and that one is corrupted/wrong that could explain it.
sha256sum ./usr/lib/libpfctl* bd164a6f18720e395fae0b30ae552afb04b5a86919e9ab8e0ba433678ef5b75b ./usr/lib/libpfctl.a aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 ./usr/lib/libpfctl.so aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 ./usr/lib/libpfctl.so.5 -
@flole said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
Maybe it's the library thats not matching? Snort doesn't call pfctl I think, if it uses the library which pfctl also uses and that one is corrupted/wrong that could explain it.
sha256sum ./usr/lib/libpfctl* bd164a6f18720e395fae0b30ae552afb04b5a86919e9ab8e0ba433678ef5b75b ./usr/lib/libpfctl.a aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 ./usr/lib/libpfctl.so aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 ./usr/lib/libpfctl.so.5As the author of the custom Snort plugin I can tell you how it works. It is making direct
ioctl()system calls for all of its operations with the packet filter firewall. Those system calls will wind up using the library. You have a corrupted installation with mismatching versions of core system libraries. -
Looks like the expected values:
[22.05-RELEASE][admin@4100-2.stevew.lan]/root: sha256 /usr/lib/libpfctl* SHA256 (/usr/lib/libpfctl.a) = bd164a6f18720e395fae0b30ae552afb04b5a86919e9ab8e0ba433678ef5b75b SHA256 (/usr/lib/libpfctl.so) = aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 SHA256 (/usr/lib/libpfctl.so.5) = aa7aa511a5d26c453dbbae04568ae5cecd39407d2b53be1fc66d51669af796c4 -
Did you by chance, at any point in the recent past, install some third-party package or otherwise switch the pkg repo pointer? The symptoms sound like some libraries on the system do not now match up with others.
-
@bmeeks said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
You have a corrupted installation with mismatching versions of core system libraries.
Not entirely sure how that could happen by simply rebooting and why the initial loading of the rules when booting works (aswell as querying the rules/tables and so on) though. To me that sounds like something after the initial loading is causing those errors, otherwise no rules would be loaded?
Others solved the same behaviour by rebooting, that obviously doesn't fix a mismatched version.
-
I suspect in the cases that fixed this by simply rebooting it had somehow loaded an old kernel from a different device.
I assume you are running Snort?
-
@bmeeks said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
Did you by chance, at any point in the recent past, install some third-party package or otherwise switch the pkg repo pointer? The symptoms sound like some libraries on the system do not now match up with others.
No I didn't, I did a normal reboot and since then things started acting in this weird way. I don't think I updated any packages since I upgraded to 2.6.0 (and that was already a few weeks ago). A messed up library would be used immediately without rebooting though.
-
@flole said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
@bmeeks said in Strange error: There were error(s) loading the rules: pfctl: pfctl_rules:
You have a corrupted installation with mismatching versions of core system libraries.
Not entirely sure how that could happen by simply rebooting and why the initial loading of the rules when booting works (aswell as querying the rules/tables and so on) though. To me that sounds like something after the initial loading is causing those errors, otherwise no rules would be loaded?
Others solved the same behaviour by rebooting, that obviously doesn't fix a mismatched version.
The Snort error you posted is coming directly from the custom blocking plug-in code. When the Snort binary is implementing a "block", it does so by making a
pfcall using theioctl()primitive within FreeBSD. It passes an Op-Code along with the system call letting the OS know what it wants. In the case of your error, it is askingpfto add an IP address to the snort2c table. That is what the DIOCRADDADDRS op-code means.The fact that call is failing tells me that the code for that operation is not present in the OS. But it should always be there.
