Domain Override results in both A record and SERVFAIL response
-
I have two pfsense systems connected together via an IPSec tunnel. Both pfsense systems run DNS (among other services) for their local networks. Each network uses the DNS name <location>.example.com for local systems. On each pfsense system I have configured the "domain overrides" to point the <other-location>.example.com to the IP address of the other location's pfsense system. From Windows clients on either network, this setup seems to work fine. I can "nslookup <host>.<other-location>.example.com and get IP address back. However, from linux clients I am getting both an IP address and SERVFAIL at the same time.
This is what I see from a linux client doing a nslookup and a dig for a host in the other location:
user@debianws:~$ nslookup pfsense.location1.example.com Server: 10.10.10.1 Address: 10.10.10.1#53 Non-authoritative answer: Name: pfsense.location1.example.com Address: 10.20.1.1 ** server can't find pfsense.location1.example.com: SERVFAIL user@debianws:~$ dig pfsense.location1.example.com ; <<>> DiG 9.16.50-Debian <<>> pfsense.location1.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62716 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;pfsense.location1.example.com. IN A ;; ANSWER SECTION: pfsense.location1.example.com. 2960 IN A 10.20.1.1 ;; Query time: 0 msec ;; SERVER: 10.10.10.1#53(10.10.10.1) ;; WHEN: Sun Sep 22 14:59:30 MST 2024 ;; MSG SIZE rcvd: 75Note the IP returned (10.20.1.1) is correct.
If I query for a name in the same local network, I get back a response that does not include the SERVFAIL:
user@debianws:~$ nslookup pfsense.location2.example.com Server: 10.10.10.1 Address: 10.10.10.1#53 Name: pfsense.location2.example.com Address: 10.10.1.1 user@debianws:~$ dig pfsense.location2.example.com ; <<>> DiG 9.16.50-Debian <<>> pfsense.location2.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48010 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;pfsense.location2.example.com. IN A ;; ANSWER SECTION: pfsense.location2.example.com. 3600 IN A 10.10.1.1 ;; Query time: 0 msec ;; SERVER: 10.10.10.1#53(10.10.10.1) ;; WHEN: Sun Sep 22 15:03:47 MST 2024 ;; MSG SIZE rcvd: 72I assume the nslookup result is because it is doing both a A record lookup (which works) and a AAAA record lookup (which is what fails). But why does this only happen when doing nslookup of a name covered by a domain override and NOT happen when querying a name handled by the local pfsense? What is going on here?
-
So I think I found my issue -- the authoritative server (the DNS server at the other location) returns a "no answer" for the AAAA record since one doesn't exist. However, if I ask the local DNS server for that AAAA record, and it forwards the request to the DNS server at the other location, it ends up returning to the client a "SERVFAIL" response. So when the local DNS server gets a "no answer" for a record from a forwarder, it instead returns a SERVFAIL to the client -- is that expected behavior? Is there an configuration option to instead have it return the same message it got from the authoritative server (in this case, a "no answer")?
-
@rtadams89 said in Domain Override results in both A record and SERVFAIL response:
Server: 10.10.10.1
Just checking : you do not have installed pfBlockerng, right ?
-
@Gertjan no
-
@rtadams89 if your going to use nslookup and want to know exactly what is being asked and returned use debug.
nslookup unlike dig would also being sending your search suffix query. Not just AAAA
$ nslookup Default Server: sg4860.home.arpa Address: 192.168.9.253 > set debug > nas.home.arpa Server: sg4860.home.arpa Address: 192.168.9.253 ------------ Got answer: HEADER: opcode = QUERY, id = 2, rcode = SERVFAIL header flags: response, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa.home.arpa, type = A, class = IN ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 3, rcode = SERVFAIL header flags: response, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa.home.arpa, type = AAAA, class = IN ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 4, rcode = NOERROR header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 1, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa, type = A, class = IN ANSWERS: -> nas.home.arpa internet address = 192.168.9.10 ttl = 3600 (1 hour) ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 5, rcode = NOERROR header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa, type = AAAA, class = IN ------------ Name: nas.home.arpa Address: 192.168.9.10 >notice how it asked for nas.home.arpa.home.arpa where home.arpa is my search suffix
$ ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : i9-win Primary Dns Suffix . . . . . . . : home.arpa Node Type . . . . . . . . . . . . : Broadcast IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No DNS Suffix Search List. . . . . . : home.arpaif you don't want it to do that put a . on the end of your query
$ nslookup Default Server: sg4860.home.arpa Address: 192.168.9.253 > set debug > nas.home.arpa. Server: sg4860.home.arpa Address: 192.168.9.253 ------------ Got answer: HEADER: opcode = QUERY, id = 2, rcode = NOERROR header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 1, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa, type = A, class = IN ANSWERS: -> nas.home.arpa internet address = 192.168.9.10 ttl = 3600 (1 hour) ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 3, rcode = NOERROR header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: nas.home.arpa, type = AAAA, class = IN ------------ Name: nas.home.arpa Address: 192.168.9.10 > -
At the core, my question comes down to this. Given the following config:
Local pfsense: 10.10.1.1
Domain Override setup for:
Domain = "location1.example.com"
Lookup Server IP address = "10.20.1.1"Should the following two commands return the same result:
nslookup -q=AAAA somehost.location1.example.com 10.10.1.1
nslookup -q=AAAA somehost.location1.example.com 10.20.1.1? Because currently, they don't. The first returns a SERVFAIL and the second returns a "no answer".
-
Not sure where your getting that from
Server: 10.10.10.1
Address: 10.10.10.1#53;; SERVER: 10.10.10.1#53(10.10.10.1)
;; WHEN: Sun Sep 22 14:59:30 MST 2024Where did you ask 10.20.1.1 ? Or even 10.10.1.1 for that matter?
again I would suggest you do a debug your going to do nslookup. Nslookup without debug provides zero sort of info that could be using for troubleshooting anything..
If you going to do dig, then do a dig to this 10.10.1.1 and then another dig directed at 10.20.1.1
Lets see a specific query to your servers with dig with what your asking for be it AAAA for example..
Here is a common problem users have when they do a domain override.. That is a forward, when you forward if the answer is rfc1918 it would be a rebind, so you either need to disable rebind protection or you need to set the domain your doing the override for to be set as private so rfc1918 can be returned.
https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html#dns-rebinding-protections
If I was troubleshooting a domain override I was setting up, first thing I would do would be directly query the ns I am forwarding to and validate it actually answers for what I am asking for. If when I then ask pfsense for that and I don't get the answer, and the answer should be rfc1918 validate I have my private set right, and that pfsense can even use the correct interface to talk to this other NS.. Do you have outbound interface set to all, or specific interfaces?
If still having problems I would sniff on pfsense and validate it actually sent the query on to this other NS and did it get a reply? etc.
Also I would never use nslookup for really anything ;) its a horrible dns client - but if I was forced to use it to try and figure out what is actually going on.. I would set debug so can see exactly what its asking for, etc.. And you can set it to query a specific NS as well the server command
-
@johnpoz I have done exactly what you are describing. I have multiple interfaces on pfsense, all are accepting DNS requests -- 10.10.1.1/10.10.10.1/10.10.20.1/etc are all different interfaces on the same pfsense system. Here is my setup that shows exactly what I am experiencing/asking about:
Clint is 10.10.10.118/24
Its local pfsense interface is at 10.10.10.1
The remote pfsense (for the network which contains the name I'm trying to resolve) is at 10.20.1.1
The name I am trying to resolve is synology.location1.example.com (which has an A record of 10.20.20.10 and no AAAA record)nslookup against the local pfsense for AAAA record:
user@debianws:~$ nslookup -debug -q=AAAA synology.location1.example.com 10.10.10.1 Server: 10.10.10.1 Address: 10.10.10.1#53 ------------ QUESTIONS: synology.location1.example.com, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ ** server can't find synology.location1.example.com: SERVFAILnslookup against the remote pfsense for AAAA record:
user@debianws:~$ nslookup -debug -q=AAAA synology.location1.example.com 10.20.1.1 Server: 10.20.1.1 Address: 10.20.1.1#53 ------------ QUESTIONS: synology.location1.example.com, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ *** Can't find synology.location1.example.com: No answerYou can see the remote pfsense returns a "no answer" when called directly, but the local pfsense (which is configured to forward to that remote pfsense) returns "SERVFAIL". So that is my question, is this expected behavior or should the local pfsense return "no answer" to the client?
PS: this is not a issue with rebinding protection or general network connectivity. DNS requests for records that exist work fine. It's only when a client asks for a record that does not exist (AAAA records) does a problem exist. The remote pfsense is returning exactly what I would expect ("no answer") when asking it directly for such a non-existent record, but the local pfsense is returning a unexpected "SERVFAIL" instead of "no answer".
-
What zone type are you using btw, transparent (default) or static or something else? And what acls do you have set?
I tried duplicated your setup with 2 different pfsense, my normal pfsense using home.arpa, my 2nd pfsense uses test.mydomain.tld and it does the same thing with A records..
So I can ask it for a name I created on my 2nd pfsense nas.test.mydomain.tld I get an answer
$ dig @192.168.9.34 nas.test.mydomain.tld ; <<>> DiG 9.16.50 <<>> @192.168.9.34 nas.test.mydomain.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20195 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;nas.test.mydomain.tld. IN A ;; ANSWER SECTION: nas.test.mydomain.tld. 3600 IN A 10.20.30.40 ;; Query time: 2 msec ;; SERVER: 192.168.9.34#53(192.168.9.34) ;; WHEN: Mon Sep 23 14:10:51 Central Daylight Time 2024 ;; MSG SIZE rcvd: 66; <<>> DiG 9.16.50 <<>> @192.168.9.253 nas.test.mydomain.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18896 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nas.test.mydomain.tld. IN A ;; ANSWER SECTION: nas.test.mydomain.tld. 3130 IN A 10.20.30.40 ;; Query time: 0 msec ;; SERVER: 192.168.9.253#53(192.168.9.253) ;; WHEN: Mon Sep 23 14:10:18 Central Daylight Time 2024 ;; MSG SIZE rcvd: 66here is asking for that same nas fqdn with AAAA, which there is no record for
$ dig @192.168.9.34 nas.test.mydomain.tld AAAA ; <<>> DiG 9.16.50 <<>> @192.168.9.34 nas.test.mydomain.tld AAAA ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3482 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;nas.test.mydomain.tld. IN AAAA ;; Query time: 1 msec ;; SERVER: 192.168.9.34#53(192.168.9.34) ;; WHEN: Mon Sep 23 14:12:25 Central Daylight Time 2024 ;; MSG SIZE rcvd: 50$ dig @192.168.9.253 nas.test.mydomain.tld AAAA ; <<>> DiG 9.16.50 <<>> @192.168.9.253 nas.test.mydomain.tld AAAA ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53301 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nas.test.mydomain.tld. IN AAAA ;; Query time: 0 msec ;; SERVER: 192.168.9.253#53(192.168.9.253) ;; WHEN: Mon Sep 23 14:13:45 Central Daylight Time 2024 ;; MSG SIZE rcvd: 50If I ask for some record that doesn't exist I get nx from both
$ dig @192.168.9.34 nas1.test.mydomain.tld ; <<>> DiG 9.16.50 <<>> @192.168.9.34 nas1.test.mydomain.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 39172 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;nas1.test.mydomain.tld. IN A ;; Query time: 1 msec ;; SERVER: 192.168.9.34#53(192.168.9.34) ;; WHEN: Mon Sep 23 14:14:31 Central Daylight Time 2024 ;; MSG SIZE rcvd: 51 $ dig @192.168.9.253 nas1.test.mydomain.tld ; <<>> DiG 9.16.50 <<>> @192.168.9.253 nas1.test.mydomain.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 42385 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nas1.test.mydomain.tld. IN A ;; Query time: 6 msec ;; SERVER: 192.168.9.253#53(192.168.9.253) ;; WHEN: Mon Sep 23 14:14:44 Central Daylight Time 2024 ;; MSG SIZE rcvd: 51I am using static as my zone type, if I typo something in my own local domain I sure don't want unbound trying to resolve that. And I am using an allow acl on the 2nd pfsense, and on my normal one I am using full snoop acl.
I normally use this setting in my pfsense, because I have no desire to go look up AAAA and no devices currently have IPv6.. I only turn that on for testing, and when I do I turn off that setting.
#private-address: ::/0 # filters out all AAAA !
Its currently turned off because testing your AAAA query, let me turn it back on. Nope no change.. Let me create an AAAA record in the 2nd pfsense and see what happens when I query that from my pfsense that has no AAAA set.
-
@johnpoz I'm currently set as "transparent" on both pfsense systems. I've just tried changing both to "static" and see no change in behavior. I also tried "type transparent" and this seemed to resolve the issue. I'm not sure I understand why that is though....
-
Type Transparent:
Similar to Transparent but it also passes through queries where the name matches but the type does not. For example, if a client queries for an AAAA record but only an A record exists, the AAAA query is passed on rather than resulting in a negative response.
Static should work too.. I use static..
I would have to look a bit closer, your using just sub domain for your delegation, domain override right.. So like your 1st pfsense is example.com and your 2 ns is location1.example.com ?
-
@johnpoz local pfsense is in the domain "location2.example.com". Remote pfsense is in "location1.example.com". I have a domain override set up on local pfsense for "location1.example.com" to use the IP of the remote pfsense.
To confirm, "static" does not work, only "type transparent" works. With any of the settings, the remote pfsense returns a "no answer" for AAAA requests, but only when the remote pfsense is set to "type transparent" does the local pfsense return a "no answer" back to the client instead of a SERVFAIL. I suppose the underlying difference is with "type transparent", the remote pfsense is passing the request on to its upstream DNS servers, whereas with "static" or "transparent" it is answering directly; but I don't see how that should matter as the answer back to the local pfsense is the same in all cases.
-
@rtadams89 I could see a case for servfail from the 1st pfsense to the client.. Because it wasn't able to lookup what was asked for.
What version of pfsense are you using btw. Could be some change in the version of unbound on it.. In my test the 1st pfsense is 24.03, 2nd pfsense was just a 2.7.2 vm..
24.03 is running 1.19.3 of unbound
2.7.2 is on 1.19.1If the answer is truely passed on, you should get back an NX and SOA.
;; AUTHORITY SECTION: example.com. 1800 IN SOA ns.icann.org. noc.dns.icann.org. 2024081420 7200 3600 1209600 3600What happens when you actually query public NS for whatever that fqdn your asking your other local NS?
Where do the 2 NS forward to or do they resolve for stuff that is not a local resource.. That could come into play.. In my test that 2nd ns just resolves if their is no local record, same as my 1st ns.. And I am set to static, so if no local resource in its domain it would just send back nx
-
@johnpoz Both pfsense instances are 2.7.2-RELEASE
Both pfsense instances are setup with 8.8.8.8 and 1.1.1.1 as their DNS servers
-
Well seems like you have it working how you want with type transparent. But to be honest what do you care gets returned with some AAAA query, if you have no AAAA records?
If you have not AAAA, do you even use IPv6 externally? If not I would just turn off answering any AAAA with the command I posted above.
I even turned off AAAA in my browser, because its stupid to ask for AAAA record if you don't even have an IPv6 address ;)
-
@johnpoz I hit an edge case. I have some uptime monitoring software which takes a hostname as input. It resolves that hostname to both A and AAAA address (this is not configurable) and then attempts to connect to the returned IPv4/IPv6 addresses. If it gets back "no answer" it figures there is no IPv4/IPv6 record and just does not try to connect to that address. However, if it gets back a SERVFAIL or other error, it figures there is a problem and reports that the hostname it is monitoring is down.
-
Hi,
I'm experiencing the same issue:- Two pfSense boxes connected via IPSEC, let's call them P1 an P2
- P2 has a domain ovverride set up so that it will query P1
- If a query B2 for an host on the overridden domain, I get the expected reply for the "A" record and an error for the "AAAA" record
- If a query B1 for the same, I get the expected reply for the "A" record and empty result for the "AAAA" record
I've attached:
- DNS status screenshot after the query of P2
- tcpdump output on P1 showing the dns requests
- dns reply on wireshark
It seems that empty answers are treated like timed out answers.
Is this the expected behaviour?
-
@rtadams89 Is it the check-mk agent? Have you managed to solve?