Authoritative DNS server behind resolver (unbound)
-
Happy holiday weekend everyone!
I have incoming port 53 NAT'ed to an internal DNS server which is publicly authoritative for several domains. This is working fine.
I have the resolver (unbound) answering DNS queries for everyone behind pfSense. This is working fine.
What isn't working fine is when these two scenarios intersect. When a client issues a DNS query to the resolver for a domain (let's say contoso.com) on the internal DNS server the resolver returns a SERVFAULT error.
If I add contoso.com to the resolver's domain overrides it works. But this is not ideal because there are many domains on this DNS server and they change frequently.
See the attached images for visualizations of these scenarios and what the desired outcome should be.
Is my desired outcome not reasonable? If it is, how do I go about getting the resolver to do this?
Thanks!
-
Correction: The resolver returns SERVFAIL, not SERVFAULT
Also: Firewall rules are wide open with allow-any-to-any on both private networks. (This will not be the end result, but for testing I have everything allowed). Right now everyone is able to talk to each other directly without issue.
-
And how exactly you imagine this "NAT'ed back" should work? The "public IP" DNS server send Unbound back to itself for the domains about which it has no information whatsoever, "surprisingly" you get SERVFAIL…
-
If someone on the internet does a DNS query for contoso.com the public IP of pfSense is returned as the nameserver. That public IP is then queried for the IP address of contoso.com, which correctly gets NAT'ed back into the DNS server. This is all well and good.
I would expect the client to see the same results from a private interface. If the client does a DNS query for contoso.com they should also get back the public IP of pfSense as the nameserver, and when that public IP is queried for the IP address of contoso.com it should therefore arrive on the public IP interface and get NAT'ed exactly the same way as any other internet client does.
At least that's how I imagine it should work, and in fact that's exactly how it does work if I enable forwarding mode on the resolver. But for some reason the resolver won't do that itself - I have to forward the entire lookup process to an external resolver for it to work. This doesn't seem like the correct behavior, but maybe I'm wrong. Or maybe I just don't have the resolver configured correctly?
-
Here's another piece of evidence:
When I dig to the public WAN interface of pfSense I can resolve contoso.com just fine from the private client. When I dig to the private interface of pfSense I get SERVFAIL when trying to resolve contoso.com.
So it seems the resolver in pfSense is either not discovering that it's own WAN interface is the nameserver or it is failing to drop the DNS query on that WAN interface once it does discover it.
I'm just wondering if this is expected behavior of the resolver, and if so, why?
-
Dude. Yeah, it will get correctly NATed from WAN. There's no reason to NAT anything from the host itself. Sigh.
Hey, root server, where's .com?
A.B.C.D
Hey, A.B.C.D, where's contoso.com?
That's yourself, silly, why are you asking?
He? I have no info on contoso.com. SERVFAIL.the resolver in pfSense is either not discovering that it's own WAN interface is the nameserver
Except that it's NOT. It's some other machine on LAN behind shitty NAT – which it will never learn.
-
Aha…that makes sense. pfSense is wearing 2 hats here - router and resolver. The resolver is told that it should know about contoso.com itself so the router/NAT part never gets involved after that.
Crap...that means I'm stuck using the resolver in forwarding mode.
-
Well yeah there're no inbound DNS queries to get NATed to the LAN DNS server.
-
I have the resolver set to listen to the private interfaces only. Any DNS traffic coming in on the WAN interface bypasses the resolver and is passed back to the internal DNS server.
At least that's how I assume it's working, because from the WAN side I am most definitely getting DNS results from the internal DNS server which don't exist anywhere else.
-
But that leaves me with a question of why my scenario can't work.
If the resolver is not set to listen to the WAN interface - the same interface it receives as the nameserver - why can't the resolver be smart enough to realize that?
"I've just been told that my WAN interface is the nameserver, but I'm not listening to that interface, so I better go ahead and query that interface for the DNS lookup I've been asked to resolve."
-
No, the DNS server is not "smart enough" to start asking a DNS server it doesn't know about. Once you've written one with such paranormal abilities, please let us know. And yes, you are getting replies when querying from WAN because there's a traffic to match your NAT rule. There's just damn nothing to NAT with outbound DNS. It's completely opposite direction.
Unbound random port -> WAN DNS port 53
WAN DNS port 53 -> Unbound random portWhere on earth does that match your NAT?
-
No, the DNS server is not "smart enough" to start asking a DNS server it doesn't know about.
Not the DNS server - the DNS resolver. Those are two different things, right? My pfSense box has no DNS server on it - only the resolver (unbound).
There's just damn nothing to NAT with outbound DNS.
I don't understand where outbound NAT comes into this. My outbound NAT settings are wide open on both subnets - allow any to any.
From the private client machine, the following commands work:
- dig @pfSensePrivateIP pfsense.org ## pfSense resolver to exterior nameserver
- dig @pfSenseWanIP contoso.com ## bypass pfSense resolver directly to NAT'ed interior nameserver
- dig @8.8.8.8 contoso.com ## exterior resolver to NAT'ed interior nameserver
This command doesn't work:
- dig @pfSensePrivateIP contoso.com ## use pfSense resolver to NAT'ed interior nameserver
Command 3 above shows that the expected nameserver is being returned as authoritative for contoso.com and that incoming DNS queries are reaching the internal authoritative DNS server. To rule out caching I added a new temporary A record and confirmed that it was being resolved by this command.
Command 2 above shows that the expected path is open and works - the client is able to get its DNS query past the firewall and onto the public interface. The public interface subsequently is able to NAT the DNS query internally to the authoritative DNS server for name resolution. For some reason the DNS resolver on pfSense is either not willing or not able to do this same thing on behalf of the client.
-
Probably because NAT reflection is a dirty, nasty hack.
- dig @pfSensePrivateIP pfsense.org ## pfSense resolver to exterior nameserver
DNS query originating from LAN to an external nameserver. Resolver on pfSense asks external nameserver for answer and forwards the response
- dig @pfSenseWanIP contoso.com ## bypass pfSense resolver directly to NAT'ed interior nameserver
NAT reflection forwards request from LAN back to DNS server.
- dig @8.8.8.8 contoso.com ## exterior resolver to NAT'ed interior nameserver
External resolver queries WAN, gets port forwarded to local name server and receives reply
- dig @pfSensePrivateIP contoso.com ## use pfSense resolver to NAT'ed interior nameserver
This one's different. You are asking the pfSense resolver to query the NAT-reflected WAN IP address. But this time the request is not coming from LAN, it is coming from pfSense itself (the resolver) (You ask the resolver for an answer, the resolver makes queries on your behalf).
I don't know enough about NAT reflection to tell you why it doesn't work.
You have your answer - domain overrides in the resolver. You say that's too much work. Not quite sure what to tell you.
Put your authoritative domains on HE.net or dyn or both and stop mucking around with NAT and, in particular, NAT reflection.
Put another unbound resolver on your LAN and tell your LAN clients to use it instead of the one built into pfSense and it'll probably work.
-
the request is not coming from LAN, it is coming from pfSense itself (the resolver) (You ask the resolver for an answer, the resolver makes queries on your behalf).
Yeah. Which has been repeated about 20 times now but the OP completely refuses to understand.
-
Thank you Derelict.
The resolver allows me to specify an interface for outgoing queries, so the expectation is that if I specify the LAN interface it should work the same as if the query came from the LAN client itself. Obviously this isn't working as expected, and I suspect you are correct that the "dirty, nasty hack" of NAT reflection is to blame.
I appreciate your help - have a great weekend!
-
No. It has to use the WAN interface to get at the WAN IP to get NAT reflected. That's why the damn thing isn't working unless you do the overrides which tells it to NOT go out to the internet but to go out LAN to resolve the names instead.
-
I realize this is an old topic, but I recently faced the same problem as OP and would like to share an alternative solution I implemented. To illustrate:
- My pfSense gateway is gw1.contoso.com.
- gw1.contoso.com is running an unbound DNS resolver for an internal LAN. It does not use forwarding mode.
- gw1.contoso.com port forwards DNS queries from WAN to an internal DNS server authoritative for contoso.com.
- contoso.com has a nameserver record, ns1.contoso.com, that I will resolve to test DNS.
- Let the public IP for all contoso.com names be 172.200.4.4.
LAN clients are able to resolve using the WAN IP thanks to NAT redirection, however asking Unbound for contoso records results in SERVFAIL after an initial timeout. My authoritative server runs several domains in addition to contoso.com, so I'm very interested in Unbound resolving everything on behalf of internal clients rather than offloading any work to them. I also do not like the idea of maintaining two sets of DNS records; NAT reflection works fine so I'd rather just fix DNS.
I added verbosity: 3 to the custom DNS resolver settings, and the debug log indicates that Unbound does in fact send a request to find the NS address for contoso.com, finds that it resolves to 172.200.4.4, then sends a request to gw1's own WAN address, 172.200.4.4:
debug: sending to target: <ns1.contoso.com.>172.200.4.4#53</ns1.contoso.com.>Unbound is bound to 127.0.0.1 and the LAN address; by some testing, it seems packets originating from gw1 do not get processed for NAT. Since no DNS service is listening on the WAN IP, and Unbound wants to check the WAN IP, I figured the easiest/quickest workaround would be to set up the DNS forwarder service to listen on that address.
DNS Forwarder is therefore enabled and configured to bind only to WAN Interface address. I received an angry message about conflicting port 53, so I hacked /etc/inc/services.inc to always put –port=53 and entered port 54 into the web interface :)
The next step was to replace "--all-servers" with "--server=AUTHORITATIVE-SERVER --no-resolv" so dnsmasq is only used to query my own records and will still work if WAN is down. This change is totally optional if you don't care. You'll know it's set up right if you can do this:
[2.4.2-RELEASE][admin@gw1.contoso.com]/root: nslookup google.com 172.200.4.4 Server: 172.200.4.4 Address: 172.200.4.4#53 ** server can't find google.com: REFUSED [2.4.2-RELEASE][admin@gw1.contoso.com]/root: nslookup ns1.contoso.com 172.200.4.4 Server: 172.200.4.4 Address: 172.200.4.4#53 Name: ns1.contoso.com Address: 172.200.4.4Now Unbound sends requests to hacked DNS Forwarder listening on 172.200.4.4, and DNS Forwarder redirects to the authoritative DNS server. All is working. And since no firewall rules are changed, WAN DNS requests and LAN requests to WAN IP are still directly NATed to the authoritative server. Perfect.
sockstat -4 -l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS dhcpd dhcpd 31055 7 udp4 *:67 *:* nobody dnsmasq 24521 4 udp4 172.200.4.4:53 *:* nobody dnsmasq 24521 5 tcp4 172.200.4.4:53 *:* root php-fpm 21690 4 udp4 *:* *:* unbound unbound 65258 3 udp4 192.168.1.1:53 *:* unbound unbound 65258 4 tcp4 192.168.1.1:53 *:* unbound unbound 65258 5 udp4 192.168.100.1:53 *:* unbound unbound 65258 6 tcp4 192.168.100.1:53 *:* unbound unbound 65258 7 udp4 127.0.0.1:53 *:* unbound unbound 65258 8 tcp4 127.0.0.1:53 *:* unbound unbound 65258 12 tcp4 127.0.0.1:953 *:* unbound unbound 65258 26 udp4 *:11960 *:* unbound unbound 65258 27 udp4 *:25771 *:* unbound unbound 65258 28 udp4 *:51981 *:* unbound unbound 65258 30 udp4 *:7219 *:* ....Note: In reality I am running my servers in a DMZ [192.168.100.1] that LAN clients access through NAT reflection, however I did not include details previously to simplify the problem. The only extra configuration needed is to enable Pure NAT redirection for relevant fw rules, change NAT mode from automatic to hybrid, and create a NAT profile for any LAN -> DMZ traffic that is allowed by the firewall rules.
-
After updating from pfSense 2.4.3 to 2.4.4p3 my LAN clients received SERVFAIL for domains I control. According to unbound logging I was still getting a response from Dnsmasq forwarder, and dig/nslookup against forwarder worked fine, so I assume Unbound was upset about the ra flag in the response. (Dnsmasq really isn't supposed to be used this way.) Rather than hacking Dnsmasq code I have replaced dnsmasq with this script to forward Unbound's query packets right to my authoritative server. Python is available in the package manager.
https://github.com/kongluoxing/py-simple-udp-proxy
