Domain overrides not working (was working until I noticed just now)

kevindd992002

I have two sites connected via an s2s wg tunnel with dns resolver domain overrides set on both sides to resolve hosts on domains on both sides.

WG tunnel network: 10.0.3.0/29

Site 1:
WG interface IP: 10.0.3.1/29
LAN interface IP: 192.168.10.1/24
unbound settings:

listening on LAN and localhost
outgoing network interface is localhost
domain override for site2.arpa set to use 192.168.20.1 (LAN interface IP of site 2)
automatically-created ACL by WG allows 10.0.3.0/29
manual ACL allows 192.168.20.0/24 (LAN subnet of site 2)

outbound NAT:

Site 2:
WG interface IP: 10.0.3.2/29
LAN interface IP: 192.168.20.1/24
unbound settings:

listening on LAN and localhost
outgoing network interface is localhost
domain override for site1.arpa set to use 192.168.10.1 (LAN interface IP of site 1)
automatically-created ACL by WG allows 10.0.3.0/29
manual ACL allows 192.168.10.0/24 (LAN subnet of site 1)

outbound NAT:

As you can see, both sites have exactly the same settings. When resolving xxx.site2.arpa from site 1 (whether from the FW itself or from a client in the LAN network), everything works fine. However, when resolving xxx.site1.arpa from site 2 (whether from the FW itself or from a client in the LAN network), it does not work.

Here's a packet capture from site 1's WG interface that proves that the packets originating from site 2 do reach that remote interface:

13:52:34.076983 IP 10.0.3.2.26488 > 192.168.10.1.53: UDP, length 44
13:52:39.150033 IP 10.0.3.2.18887 > 192.168.10.1.53: UDP, length 44

You can see that there are no reply packets, for some odd reason. I have a separate DNS server (AGH) on site 2 that serves my LAN clients which also has a domain override to 192.168.10.1 and this works without any issues. So the domain override issue only happens when using unbound in site 2's pfsense box.

Any ideas?

kevindd992002

@johnpoz maybe you have any ideas?

kevindd992002

I just didn't specify this in my original post but I have a hub-and-spoke tunnel setup. Site 1 is the hub and site 2 is one of the spokes. Site 3 is another spoke and experiences the same exact issue when trying to resolve xxx.site1.arpa. The site 3 pfsense box has the same exact configuration I stated above. Not sure if this matters but the pfsense versions are:

site 1 = 2.6
site 2 = 2.7 dev
site 3 = 2.5.2

kevindd992002

Anyone can help me here?

msswift

@kevindd992002 All I can say is domain overrides are working fine for me across dnsmasq and openvpn site-to-site tunnels in both 2.5.2 and 2.6.0. I don't have experience with unbound or wireguard. You say your setup "was working" so something changed, and at a guess, a firewall is swallowing those replies or the route back to the querier is now incorrect. Doesn't narrow it down much I know, but each of those possibilities can be investigated with tests (log default block/pass rules, add particular rules to log, and use packet capture on Pfsense till you find those reply packets and then follow them till they go awry. If you're not sure the queries are arriving in the first place and generating reply packets, start with that leg first. Dnsmasq can log all queries whcih makes that part easy; I don't know about unbound. good luck!

kevindd992002

Ok, I've had time to work on this again and I've isolated the problem to the home.arpa domain! When the query from the site 2 and 3 pfsense boxes are directed to the home.arpa domain, unbound doesn't even use the override rule at all (proven by packet capture logs). It just simply drops the packets.

Site 1 = home.arpa
Site 2 = condo.arpa
Site 3 = jojo.arpa

If I create a domain override for any other domain on Site 2 and 3 and point it to the DNS resolver in Site 1, everything works fine! The queries from the site 2 and 3 pfsense boxes (localhost) are being forwarded just fine.

So this has to be a change on the pfsense software side disallowing the use of home.arpa domain overrides for some reason, though I'm not sure what the reason behind this is because that domain is very known to be used for, well, "home" use.

kevindd992002

Any help here please?

Bob.Dig

@kevindd992002 Then change Site1 also and look if your assumption was correct?

kevindd992002

@bob-dig change Site 1 to what? Are you suggesting changing the whole local domain of site 1?

kevindd992002

Also, the packet captures on both sites 2 and 3 do not show the packets being forwarded to the destination dns server when the domain override is home.arpa. So technically you can ignore site 1.

Bob.Dig

This post is deleted!

SteveITS

@kevindd992002 Have you added the other subnets to the Access List in site 1?
"By default, IPv4 and IPv6 networks residing on internal interfaces of this firewall are permitted. Additional networks must be allowed manually."

kevindd992002

@steveits yes, I did. Also, the wireguard service does this automatically for you. And like I said, it works when I forward to the site1 dns resolver for another domain, do ACLs are not the problem.

SteveITS

@kevindd992002 Hmm, home.arpa is a special domain (https://www.iana.org/domains/arpa) so maybe that is confusing things? I wonder if myhome.home.arpa or something like that would behave differently.

kevindd992002

@steveits said in Domain overrides not working (was working until I noticed just now):

@kevindd992002 Hmm, home.arpa is a special domain (https://www.iana.org/domains/arpa) so maybe that is confusing things? I wonder if myhome.home.arpa or something like that would behave differently.

Exactly. home.arpa is what people should be using in their home environment. That is its purpose. And I'm 100% sure this was working not too long ago so something changed with unbound in maybe 2.5.2 and above.

kevindd992002

@jimp I replied to the bug I filed here:

https://redmine.pfsense.org/issues/13065?tab=history

kevindd992002

Any other ideas here?

When I try to do a DNS Lookup on the firewall of sites 2 and 3, I don't even see the home.arpa domain being listed under Status -> DNS Resolver. This is another indication that the query is somehow being dropped if it's for the home.arpa domain.

johnpoz

@kevindd992002 said in Domain overrides not working (was working until I noticed just now):

I don't even see the home.arpa domain being listed under Status -> DNS Resolver

Well then you don't have your domain override setup... Once you setup a domain overrride it would be there all the time, ask unbound how it would lookup home.arpa.

Actually do a query and it would be listed..

Once you do a lookup - it would be listed in the dns resolver status.

And then if I ask unbound again - it would be listed in cache.

Be it that IP even answers for that domain or not, etc.. - mine sure doesn't - I don't even have 192.168.9.42 device..

jimp

As I mentioned on the Redmine entry there is nothing special about home.arpa in pfSense other than it being the default domain name under System > General Setup. When it is that domain, it has special settings in unbound automatically but if you have changed that then it wouldn't treat it any differently.

You'll need to post a lot more of your setup here. It could be any number of things. Missing routes in the routing table for the firewall itself to reach places both ways. Missing ACLs in Unbound to allow queries from the other sites. Something wrong in your unbound config or domain override. There are lots of moving parts to get this working between sites and it's even harder with WireGuard since more of it is manually managed than with other methods.

Check the routing table on each node and ensure it has routes over the appropriate WireGuard interfaces for the appropriate destinations
Check the WireGuard interface firewall rules to ensure the traffic will pass between the hosts (remember to cover both the LAN(s) and the WireGuard interface addresses)
Check if you can ping the remote firewall LAN addresses with a source of Localhost and the LAN since that's how you setup Unbound, e.g. ping -S 127.0.0.1 <other fw LAN IP address> and ping -S <this LAN IP address> <other fw LAN IP address>
Check Services > DNS Resolver, Access Lists tab and ensure there are entries there for the other firewall LANs and the WireGuard interface subnets. Some of those may be automatically added, check /var/unbound/access_lists.conf to confirm
When you ping or send traffic across, check the contents of the state table to ensure the states are on the correct interfaces with the expected addresses
Your outbound NAT rules are over-matching, they will NAT traffic out an interface with its own address, which can break some things. You have it set to port 53 but even so it's better to make sure you aren't doing it unnecessarily. Make a specific rule for localhost as a source that will NAT all outbound, not just port 53. You shouldn't need to NAT traffic from the LAN that should be handled by routing, no need for NAT.
Compare the contents of /var/unbound/host_entries.conf and /var/unbound/domainoverrides.conf and look for instances of the domains in question and ensure they match up as expected.

If all else fails, from all of the firewalls involved post the entire contents of /var/unbound/unbound.conf, /var/unbound/domainoverrides.conf, /var/unbound/host_entries.conf, /var/unbound/access_lists.conf, the output of ifconfig -a and netstat -rn along with the contents of /tmp/rules.debug (at least for the wireguard interfaces and localhost). You can redact private info as long as it's done consistently so that people can identify the same address in different places (e.g. 192.168.10.x -> xxx.xxx.xx.x, 192.168.20.x -> xxx.xxx.yy.x, and so on).

kevindd992002

@johnpoz said in Domain overrides not working (was working until I noticed just now):

@kevindd992002 said in Domain overrides not working (was working until I noticed just now):

I don't even see the home.arpa domain being listed under Status -> DNS Resolver

Well then you don't have your domain override setup... Once you setup a domain overrride it would be there all the time, ask unbound how it would lookup home.arpa.

Actually do a query and it would be listed..

Once you do a lookup - it would be listed in the dns resolver status.

And then if I ask unbound again - it would be listed in cache.

Be it that IP even answers for that domain or not, etc.. - mine sure doesn't - I don't even have 192.168.9.42 device..

I do have it though:

[2.7.0-DEVELOPMENT][root@pfSense.condo.arpa]/root: unbound-control -c /var/unbou                                                                                  nd/unbound.conf lookup home.arpa
The following name servers are used for lookup of home.arpa.
forwarding request:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 1 IP addresses.
192.168.10.1            not in infra cache.

But I still don't see anything in System -> DNS Resolver for either 192.168.10.1 or home.arpa. When I do a second lookup, it's not being put into cache too:

[2.7.0-DEVELOPMENT][root@pfSense.condo.arpa]/root: unbound-control -c /var/unbound/unbound.conf lookup home.arpa
The following name servers are used for lookup of home.arpa.
forwarding request:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 1 IP addresses.
192.168.10.1            not in infra cache.