Bind DNS Package AAAA filtering problem

[dweimer]

Bind AAAA filtering, causing error on reinstalled system

Feb 9 09:08:42 	named 	45094 	exiting (due to fatal error)
Feb 9 09:08:42 	named 	45094 	loading configuration: failure
Feb 9 09:08:42 	named 	45094 	/usr/local/lib/named/filter-aaaa.so: plugin configuration failed: failure
Feb 9 09:08:42 	named 	45094 	failed to dlopen() plugin '/usr/local/lib/named/filter-aaaa.so': /usr/local/lib/named/filter-aaaa.so: mmap of data failed: Permission denied
Feb 9 09:08:42 	named 	45094 	loading plugin '/usr/local/lib/named/filter-aaaa.so' 

I have a few systems running this setup that are working. This system was reinstalled due to a different issue and after restoring configuration I had to remove the filter-aaa plugin to get Bind DNS service to run.

I have confirmed from working systems and this system that the file exists with same permissions and ownership, only difference is date time stamp. Even used SFTP to transfer file from working system to broken one in case it was a file corruption issue.

find / -name "filter-aaaa.so" -exec ls -al {} \;
-rw-r--r--  1 root  wheel  22032 Jul 26  2021 /usr/local/lib/named/filter-aaaa.so
-rw-r--r--  1 root  wheel  22032 Feb  7 12:35 /cf/named/usr/local/lib/named/filter-aaaa.so

The /usr/local/lib/named/filter-aaaa.so file appears to get installed by the package, and is copied to the /cf/named/usr/local/lib/filter-aaaa.so file whenever a configuration change is made to DNS configs so that its in the changed root location for the bind sandbox.

I checked permissions and ownership of all folders from root to the file, all are the same from working and non working setup. Anyone have any ideas?

I found an older thread referencing this error, but it quickly switched into using unbound instead. I am not using bind specifically for this feature, but instead for advanced forwarding/slave zones for multiple VPN connections and the remote internal DNS servers.

[throwaway29]

@dweimer I have the same exact issue after trying to upgrade to 2.6 and then downgrading back to 2.5.2...

I've spent 2 days trying to figure out a solution. No luck. Something changed when doing the reinstall. Seems like everyone is using unbound and python to do this now, but it seems overly complicated. I'm just going to disable ipv6 at this point since pfsense is obviously screwing this up.

[throwaway29]

@bmeeks in this thread suggests it is a permission issue... but I tried chowning the file to the bind user and even making it executable but it still doesn't work.

https://forum.netgate.com/topic/151745/bind-filter-aaaa/2

[fitch]

@throwaway29 Just wanted to say this same exact issue happened to me. My system is setup for filtering Netflix AAAA records with bind.

I upgraded from 2.5.2 but did a fresh install because I wanted to switch to ZFS. Turns out that for me, it was ZFS that was responsible. I did another fresh install but this time selected UFS and everything works as before.

Why would ZFS have anything to do with it?

[johnpoz]

@fitch said in Bind DNS Package AAAA filtering problem:

Why would ZFS have anything to do with it?

It wouldn't...

[fitch]

@johnpoz I did fresh install with ZFS and had the issue. Then did fresh install with UFS and did not. To double check, I reran fresh install with ZFS and had problem. Finally fresh install with UFS and problem is gone. Pretty conclusive to me.

I would think ZFS should not have an effect, but it does. Can it affect permissions on chrooted applications like bind? Might the installer be doing something different when the choice is ZFS? How does one figure this out?

[fitch]

@throwaway29 were all your systems ZFS or UFS?

[johnpoz]

@fitch sorry but there is just no way that the file system has anything to do with what your issue is..

So clean install means what.. Install pfsense and setup xyz for your bind aaaa filter, or are you loading some config?

Please link to or provide exact instructions on how to enable your AAAA filtering and I will be happy to try and duplicate this.. Because there is no freaking way that the file system be it ufs or zfs or btrfs or ext2 or ext4 or xfs or even ReiserFS has anything to do with it..

[fitch]

@johnpoz OK Mister "no freaking way that the file system be it ufs or zfs or btrfs or ext2 or ext4 or xfs or even ReiserFS has anything to do with it.."

I have attached a very minimal config file. I setup a vm using vmware workstation pro 16.2. I hope you have this available, but if not you may need to adjust the NIC device names. For my vm, em0 is the WAN and em1 is the LAN.

I created the vm and installed from iso file, but for my real system I booted from a USB stick. The behavior is the same.

First, install the system and choose ZFS. Just zip through the wizard, you don't need to change anything. Restoring from the config file will do everything needed. Now, there is a bug in the restore where Bind won't get reloaded because it is looking for named, but after you restore the config file just go to the package manager and install Bind.

In the dashboard, add a Service Status widget so you can see the state of named.

With ZFS chosen as the file system, you will find that named will not start. You can try to start it, but it cannot find filter-aaaa.so as described in the OP post at the top.

Now, reinstall the vm but choose UFS as the file system. Restore the config file, install Bind and low and behold named now starts.

Let me know when you have verified this.

minimal_config.xml

[johnpoz]

@fitch wasn't really looking for a xml - what I was looking for how to do your AAAA filtering from a clean setup.. And then I will take my config and then change the file system to zfs..

where Bind won't get reloaded because it is looking for named

Your issue could for sure be related to something like that..

[dweimer]

@johnpoz, @fitch, It shouldn't I have full blown FreeBSD installs running Bind with the plugin working in chrooted environment on ZFS without a problem.
However I did check, and my two working systems are running on UFS. The Broken system is on ZFS.

[johnpoz]

@dweimer wasn't the whole filter-aaaa stuff deprecated.

So if someone could provide some steps to getting it working with current bind that is part of 2.6, I will then try duplicating that after changing it to zfs.

There should be no possible way that the file system is the problem here, but there could be something missing that is causing the problem..

[dweimer]

@johnpoz There isn't a check box to enable it or anything, you have to add it to the custom options under the View configuration to enable it. Yes they are pushing to disable removing IPv6 responses in order to get more people using IPv6. However when you are stuck with a local ISP that doesn't support IPv6. I have found it problematic having remote VPN clients that do have IPv6 support not getting correct routing of traffic that I want on VPN. Due to them taking direct IPv6 path instead of resolving IPv4 address that would have been routed over VPN.

plugin query "/usr/local/lib/named/filter-aaaa.so" {
        filter-aaaa-on-v4 yes;
        filter-aaaa-on-v6 yes;
        filter-aaaa { any; };
};

[fitch]

@johnpoz you can just import that portion from the xml?

[fitch]

@johnpoz Sorry, I misunderstood. I thought you wanted a way to recreate what I had. To add Bind to your system, it is easy.
Here are instructions on how to setup the filtering via Bind the way I do it. I am basically setting Unbound to forward DNS requests for certain domains to Bind, where Bind only returns ipv4 addresses.

Install Bind:
Set things up as follows:

Settings for Unbound:
Change IP address to your LAN address. 127.0.0.1 might work but I have not tried that.

[dweimer]

@fitch & @johnpoz It is absolutely the ZFS file system. I built two VMs today one with ZFS and one with UFS (v2.6.0). I confirmed what was already reported UFS one worked ZFS one didn't. And I found out why and a way to make it work on the ZFS system.

zfs get exec pfSense/cf

The install sets this value to off, if you change it to on it works.

zfs set exec=on pfSense/cf

There is of course a security based reason this was set, and this undoes that. But the filesystem is disallowing the execution of the bind plugin even though the file level permissions are correct.

[johnpoz]

@dweimer said in Bind DNS Package AAAA filtering problem:

zfs set exec=on pfSense/cf

That seems like a bug, since bind runs in a chroot - but that is not really a "filesystem" problem that is a problem with a setting pfsense on where the chroot environment when it sets it up. Which is under cf.

I would think that would cause issues with anything running under chroot that needs to exec a file..

Should be put in as bug, so they could fix it..

I would think its something like this bug
https://redmine.pfsense.org/issues/10413

Where the file was not copied into the chroot.

I would "think" I am by no means a jail or chroot guy - would your setting of the exec on cf/named would allow for the aaaa to also work..

I show these as the settings

[22.01-RELEASE][admin@sg4860.local.lan]/: zfs get exec
NAME                  PROPERTY  VALUE  SOURCE
pfSense               exec      on     default
pfSense/ROOT          exec      on     default
pfSense/ROOT/default  exec      on     default
pfSense/cf            exec      off    local
pfSense/home          exec      on     default
pfSense/reservation   exec      on     default
pfSense/tmp           exec      on     local
pfSense/var           exec      on     default
pfSense/var/cache     exec      off    local
pfSense/var/db        exec      off    local
pfSense/var/log       exec      off    local
pfSense/var/tmp       exec      on     default

They should prob setup something specific for bind so that its plugins can work. But doesn't seem to be defined.

[dweimer]

@johnpoz I went ahead and created a bug report.

https://redmine.pfsense.org/issues/12869

[dweimer]

@johnpoz Just an FYI I went back and checked through release notes back to where the ZFS option was added to installer. I don't see any reference to when the separate /cf filesystem was added I have an older install on ZFS (that is updated all the way to 2.6.0) that wasn't using Bind plugin. It doesn't have that as a separate filesystem Bind AAAA plugin worked after installing it. I was hoping to update the bug as to which versions were effected.

[johnpoz]

@dweimer Yeah they changed the layout a few times for zfs. I know that the old 21.05.x layout was different than 22.01 that is for sure - there is a thread around here somewhere talking about that..

I would "guess" the same sort of changes happened with 2.5.2 to 2.6..

And I think the layout was different before that as well - there at least a couple of different changes to the zfs layout.. I recall I was going to do a clean install of say 21.05.2 but there was something mentioned somewhere around here that zfs was going to change again in the then 21.09 which got pushed to 22.01 so I didn't bother doing clean for 21.05.2 etc. and just upgraded since figured when 21.09 came out I would do clean for the new layout.

This is a good catch for sure - and sure will make many people happy when corrected.. Prob a few people scratching their heads on this one.. Why and the F is not working - and could see how zfs was the most likely culprit.. But I still stand by my statement that is not an actual file system problem ;)

You could have the same sort of problem with how UFS stuff is mounted read only for example.. Not sure if you can set exec permissions via UFS?

I will have to take a look at your bug - did you list the full zfs exec set showing cf set to off? I would think they would either have to change that setting, or I assume change the zfs layout yet again? ;)

I wonder if might be a good idea on the widget to list exec or not? Or option to show that in the widget?

edit: I linked to this thread in your bug report. They always like to see discussion and if others are seeing the problem, etc..