DNS Resolver domain override issue for just one client in the same network

kevindd992002

So I have two networks (one in each location) connected through a Wireguard s2s VPN.

Network 1: 192.168.10.0/24 (domain is home.arpa)
Network 2: 192.168.20.0/24 (domain is condo.arpa)

Let's concentrate on Network 2 as that's where the client that's having the issue is. Basically, the DNS resolver of the pfsense box in network 2 has a domain override for the "home.arpa" to forward all requests to 192.168.10.1 (DNS resolver of network 1).

client 1: 192.168.20.100 (pihole) -> works just fine when trying to resolve a home.arpa record
client 2: 192.168.20.200 ~ 192.168.20.254 (DHCP user pool) -> works just fine when trying to resolve a home.arpa record
client 3: 192.168.20.22 (Debian box) -> does not receive any answers when trying to resolve a home.arpa record; other external domain queries work just fine

Here are some sample dig queries from client 3:

root@epsilon:~# dig pfsense.home.arpa

; <<>> DiG 9.16.15-Debian <<>> pfsense.home.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 8498
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pfsense.home.arpa.             IN      A

;; Query time: 0 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)
;; WHEN: Tue Oct 05 22:18:27 PST 2021
;; MSG SIZE  rcvd: 46

root@epsilon:~# dig www.google.com

; <<>> DiG 9.16.15-Debian <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31722
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         300     IN      A       142.250.204.68

;; Query time: 247 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)
;; WHEN: Tue Oct 05 22:18:32 PST 2021
;; MSG SIZE  rcvd: 59

root@epsilon:~# dig www.yahoo.com

; <<>> DiG 9.16.15-Debian <<>> www.yahoo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55811
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.yahoo.com.                 IN      A

;; ANSWER SECTION:
www.yahoo.com.          59      IN      CNAME   new-fp-shed.wg1.b.yahoo.com.
new-fp-shed.wg1.b.yahoo.com. 60 IN      A       202.165.107.50
new-fp-shed.wg1.b.yahoo.com. 60 IN      A       202.165.107.49

;; Query time: 479 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)
;; WHEN: Tue Oct 05 22:18:36 PST 2021
;; MSG SIZE  rcvd: 106

As you can see, when I try querying pfsense.home.arpa, it does not return an answer. When capturing packets in my LAN interface, I do see queries from client 3 to 192.168.20.1:53 but they are not forwarded to 192.168.10.1:53, for some odd reason.

Thoughts?

johnpoz

@kevindd992002 hmmm .arpa is new special use tld - there might be something that keeps it from being forwarded.. Let me look at the hidden config..

One thing for sure that could be a problem - is when you setup a domain override, if the answer comes back as rfc1918 that would be considered a rebind and would fail unless you set the domain as private.

kevindd992002

@johnpoz but why would the issue only happening for a specific client/IP address? I've been using .arpa for a few years now as it's the recommended domain for local networks.

johnpoz

@kevindd992002 see my edit.. The problem could be rebind.. On a work meeting, kind of paying attention too ;) so haven't looked to deep into your post and examples ;) Let me get these meetings out of way and can pay more attention - heheh

edit: other than comes into play with forward and unbound is ACL.. but again need to look deeper into the info you provided ;)

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 see my edit.. The problem could be rebind.. On a work meeting, kind of paying attention too ;) so haven't looked to deep into your post and examples ;) Let me get these meetings out of way and can pay more attention - heheh

If it's rebind though, the issue should happen for all the clients in the 192.168.20.0/24 network :) Also, if it's rebind, I should at least see traffic when capturing packets in the wireguard s2s interface. But I'm not seeing traffic in that interface when the source is client 3. If the source is NOT client 3, it works just fine.

Sure thing, I'll wait for you to read my OP.

kevindd992002

Other dig examples:

root@epsilon:~# dig @192.168.20.1 pfsense.home.arpa

; <<>> DiG 9.16.15-Debian <<>> @192.168.20.1 pfsense.home.arpa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35967
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pfsense.home.arpa.             IN      A

;; Query time: 0 msec
;; SERVER: 192.168.20.1#53(192.168.20.1)
;; WHEN: Tue Oct 05 23:19:37 PST 2021
;; MSG SIZE  rcvd: 46

root@epsilon:~# dig @192.168.10.1 pfsense.home.arpa

; <<>> DiG 9.16.15-Debian <<>> @192.168.10.1 pfsense.home.arpa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30436
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pfsense.home.arpa.             IN      A

;; ANSWER SECTION:
pfsense.home.arpa.      3600    IN      A       192.168.10.1

;; Query time: 3 msec
;; SERVER: 192.168.10.1#53(192.168.10.1)
;; WHEN: Tue Oct 05 23:19:44 PST 2021
;; MSG SIZE  rcvd: 62

root@epsilon:~# dig @192.168.20.100 pfsense.home.arpa

; <<>> DiG 9.16.15-Debian <<>> @192.168.20.100 pfsense.home.arpa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63627
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pfsense.home.arpa.             IN      A

;; ANSWER SECTION:
pfsense.home.arpa.      3600    IN      A       192.168.10.1

;; Query time: 7 msec
;; SERVER: 192.168.20.100#53(192.168.20.100)
;; WHEN: Tue Oct 05 23:19:56 PST 2021
;; MSG SIZE  rcvd: 62

If client 3 uses other DNS servers (say network 2's pihole - 192.168.20.100 or directly network 1's DNS resolver), everything works fine. As soon as it queries network 2's DNS resolver, it doesn't receive an answer.

johnpoz

@kevindd992002 ok let me make sure I get this right..

You have clients in 192.168.20.x

Pfsense IP is 192.168.20.1

If 192.168.20.A asks 192.168.20.1 for something.home.arpa it works, and resolves this 192.168.10.x address for what it was looking for.

But if client 192.168.20.B asks 192.168.20.1 for the same something.home.arpa it fails with servfail?

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 ok let me make sure I get this right..

You have clients in 192.168.20.x

Pfsense IP is 192.168.20.1

If 192.168.20.A asks 192.168.20.1 for something.home.arpa it works, and resolves this 192.168.10.x address for what it was looking for.

But if client 192.168.20.B asks 192.168.20.1 for the same something.home.arpa it fails with servfail?

You got it!

johnpoz

@kevindd992002 well off the top that doesn't make any sense at all.. For starters client B should just get back the cached item..

If you do it from A, and get an answer and then right away ask from B it fails with servfail?

But B can ask 20.1 for other stuff directly.. And that all works, as long as its not in home.arpa? Well that odd AF! hmmmmm

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 well off the top that doesn't make any sense at all.. For starters client B should just get back the cached item..

If you do it from A, and get an answer and then right away ask from B it fails with servfail?

But B can ask 20.1 for other stuff directly.. And that all works, as long as its not in home.arpa? Well that odd AF! hmmmmm

Exactly, it doesn't make sense at all.

Yes, it still fails with SERVFAIL when I do that test.

Yes, the issue with B is only when asking for a home.arpa record. Other stuff (public DNS records) work fine.

Odd indeed. I'm scratching my head with this issue. I rebooted both the client and the pfsense firewall just for the heck of it and nothing changed.

kevindd992002

Here's some logs from 192.168.20.1:

Oct 5 23:48:39 	unbound 	77902 	[77902:3] info: validator operate: query pfsense.home.arpa. A IN
Oct 5 23:48:39 	unbound 	77902 	[77902:3] debug: validator[module 0] operate: extstate:module_wait_module event:module_event_moddone
Oct 5 23:48:39 	unbound 	77902 	[77902:3] debug: return error response SERVFAIL
Oct 5 23:48:39 	unbound 	77902 	[77902:3] debug: configured stub or forward servers failed -- returning SERVFAIL
Oct 5 23:48:39 	unbound 	77902 	[77902:3] info: processQueryTargets: pfsense.home.arpa. A IN
Oct 5 23:48:39 	unbound 	77902 	[77902:3] info: error sending query to auth server 192.168.10.1 port 53
Oct 5 23:48:39 	unbound 	77902 	[77902:3] notice: remote address is 192.168.10.1 port 53
Oct 5 23:48:39 	unbound 	77902 	[77902:3] notice: sendto failed: Can't assign requested address
Oct 5 23:48:39 	unbound 	77902 	[77902:3] debug: sending to target: <home.arpa.> 192.168.10.1#53
Oct 5 23:48:39 	unbound 	77902 	[77902:3] info: sending query: pfsense.home.arpa. A IN

kevindd992002

OH WAIT! It looks like even A is not working. A is pihole in this case and I forgot that I had configured pihole to forward requests directly to 192.168.10.1 for home.arpa, therefore bypassing the DNS resolver in pfsense. So at least now I know that the behavior is consistent for all 192.168.20.0/24 clients. The pfsense DNS resolver is definitely acting up at this point.

kevindd992002

One important question here. When unbound queries a request for the domain override to the destination DNS server, what source IP does it use? localhost? Because I might need to re-enable this SNAT rule to fix this:

kevindd992002

Ok, so re-enabling that SNAT rule on both sides and adding the WIREGUARD network to the DNS resolver access list on both sides fixed the problem.

johnpoz

@kevindd992002 said in DNS Resolver domain override issue for just one client in the same network:

behavior is consistent for all 192.168.20.0/24 clients.

That makes more sense.. Well depends on what natting your doing or not doing.. And yes through any forward with an domain override you can run into 2 issues.

if answer comesback as rfc1918 it would be considered a rebind - so you need to make sure the domain is set as private in the unbound doing the query.

On the other end where your forwarding too, that unbound ACL would have to be setup to allow the query from this remote network.. Sure you could work around the ACL issue with a sourcenat on that site, but better would be is just update the ACL.

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 said in DNS Resolver domain override issue for just one client in the same network:

behavior is consistent for all 192.168.20.0/24 clients.

That makes more sense.. Well depends on what natting your doing or not doing.. And yes through any forward with an domain override you can run into 2 issues.

if answer comesback as rfc1918 it would be considered a rebind - so you need to make sure the domain is set as private in the unbound doing the query.

On the other end where your forwarding too, that unbound ACL would have to be setup to allow the query from this remote network.. Sure you could work around the ACL issue with a sourcenat on that site, but better would be is just update the ACL.

The answers are definitely RFC1918's but they're still not considered as rebind. Do you know why? Of course, I want rebind protection to work.

And how do I set the domain to private in the unbound that's doing the query?

For the unbound ACL where I'm forwarding to, if I update its ACL to allow the query from the unbound that's doing the query, what source IP will I put there if I don't do sourceNAT? If I don't do sourceNAT, I won't know what the source IP of the unbound from either side because IIRC the source for unbound requests is "This Firewall" and I'm not 100% sure what that is but that's the one in my SNAT rule now and it works.

johnpoz

@kevindd992002 said in DNS Resolver domain override issue for just one client in the same network:

RFC1918's but they're still not considered as rebind. Do you know why? Of course,

Did you set them as private in the unbound config? I guess there could be some config that says anything home.arpa the like could be non rebind, but I doubt that..

Did you turn off rebind completely?

https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html

edit
As to what IP.. With default outbound nat on, it would be the query unbound pfsense IP on the tunnel network you have setup for the s2s I would believe.. If you have messed with outbound nat from auto. It would have to be an IP that pfsense could use to talk to this other network with.. That would have to be the IP it has on the s2s connection. Or it would have to be natted to that IP, etc.

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 said in DNS Resolver domain override issue for just one client in the same network:

RFC1918's but they're still not considered as rebind. Do you know why? Of course,

Did you set them as private in the unbound config? I guess there could be some config that says anything home.arpa the like could be non rebind, but I doubt that..

Did you turn off rebind completely?

https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html

edit
As to what IP.. With default outbound nat on, it would be the query unbound pfsense IP on the tunnel network you have setup for the s2s I would believe.. If you have messed with outbound nat from auto. It would have to be an IP that pfsense could use to talk to this other network with.. That would have to be the IP it has on the s2s connection. Or it would have to be natted to that IP, etc.

I did not:

johnpoz

@kevindd992002 ha - they seemed to have changed it to help users doing local forwarding.

I just added a couple of test domain forwards for testing to a local ns.. And look what gets added to the conf ;)

I do not recall seeing this in the release notes? But there it is.. look in your

[21.05.1-RELEASE][admin@sg4860.local.lan]/: cat /var/unbound/unbound.conf

I wonder when that got added - I am pretty freaking sure it didn't use to do that..

edit: Well F me - looks like that was added sometime back in 2017 from looking through the github code for unbound.inc..

kevindd992002

@johnpoz said in DNS Resolver domain override issue for just one client in the same network:

@kevindd992002 ha - they seemed to have changed it to help users doing local forwarding.

I just added a couple of test domain forwards for testing to a local ns.. And look what gets added to the conf ;)

I do not recall seeing this in the release notes? But there it is.. look in your
[21.05.1-RELEASE][admin@sg4860.local.lan]/: cat /var/unbound/unbound.conf
I wonder when that got added - I am pretty freaking sure it didn't use to do that..

edit: Well F me - looks like that was added sometime back in 2017 from looking through the github code for unbound.inc..

Lol, that makes total sense th!en. Thanks for the help!