SOA record - unable to look up using external nameserver

toskium · Mar 19, 2024, 6:50 AM

Hello,

I am experiencing difficulties in resolving SOA records when specifying an external nameserver, even from pfSense shell.

I am currently running pfSense 23.05.1-RELEASE on a Netgate 7100.

This is the command I run:

dig @1.0.0.1 SOA heise.de

The error I get is:

;; communications error to 1.0.0.1#53: timed out

When the pfSense DNS resolver is used I do get an instant reply:

dig SOA heise.de

; <<>> DiG 9.18.24-1-Debian <<>> SOA heise.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4550
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;heise.de.                      IN      SOA

;; ANSWER SECTION:
heise.de.               41081   IN      SOA     ns.heise.de. hostmaster.heise.de. 2024031902 14400 3600 604800 86400

;; Query time: 4 msec
;; SERVER: 192.168.10.1#53(192.168.10.1) (UDP)
;; WHEN: Tue Mar 19 07:41:14 CET 2024
;; MSG SIZE  rcvd: 87

Other DNS queries are working fine when specifying an external nameserver

dig @1.0.0.1 heise.de

; <<>> DiG 9.18.24-1-Debian <<>> @1.0.0.1 heise.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60506
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;heise.de.                      IN      A

;; ANSWER SECTION:
heise.de.               83910   IN      A       193.99.144.80

;; Query time: 8 msec
;; SERVER: 1.0.0.1#53(1.0.0.1) (UDP)
;; WHEN: Tue Mar 19 07:43:41 CET 2024
;; MSG SIZE  rcvd: 53

Telnet to 1.0.0.1 on port 53 is also working fine.

The setup looks like this:

2 WAN connections
2 Gateway groups (WAN1 Tier 1, WAN2 Tier 5)(WAN2 Tier 1, WAN1 Tier 5)
Default GW is set to one of the gateway groups
Firewall Rules are actively setting gateways depending on source ip address (specific hosts need to use specific gateway).
login-to-view
login-to-view

DNS Resolver is activated and configured like this:
login-to-view

Additionally to this I am running the pfblocker-NG addon, but checking the logs for blocked communication from my host to 1.0.0.1 did not show any hits in the logs.

I hope someone has an idea how to solve this issue. Thank you in advance.

Gertjan · Mar 19, 2024, 9:24 AM

@toskium said in SOA record - unable to look up using external nameserver:

dig @1.0.0.1 SOA heise.de

Who is 10.0.0.1 ? (pfSense LAN ?) edit : that's 1.0.0.1, a public DNS resolver, not 10.0.0.1, which would be local. I've miss read.
On what device are you running this this command ? (a debian box ?)
On a pfSense LAN ?

If "10.0.0.1" is pfSense, you could go here Services > DNS Resolver > Advanced Settings and set the Log Level to at least "3". Now, you can see the DNS requests for a SOA coming in, and see what unbound does with it.

@toskium said in SOA record - unable to look up using external nameserver:

; <<>> DiG 9.18.24-1-Debian <<>> SOA heise.de

Ah, ok, a Debian based device.
What are the DNS server(s) of this device ?

@toskium said in SOA record - unable to look up using external nameserver:

Telnet to 1.0.0.1 on port 53 is also working fine.

Telnet uses TCP.
DNS requests most often, if they are small enough, UDP.

Not related , but : your are forwarding, you should disable DNSSEC.

johnpoz · Mar 19, 2024, 8:25 AM

@toskium well not being able to talk to 1.0.0.1 (cloudflare on udp 53) could be some other rule you have? What about say quad9 or googledns? Or their normal 1.1.1.1 IP Also you show a timeout talking to them.. So it would be not just SOA..

As mentioned that telnet test would be tcp.. So do a dig via tcp.

$ dig @1.0.0.1 heise.de SOA +tcp

; <<>> DiG 9.16.48 <<>> @1.0.0.1 heise.de SOA +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31957
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;heise.de.                      IN      SOA

;; ANSWER SECTION:
heise.de.               86400   IN      SOA     ns.heise.de. hostmaster.heise.de. 2024031918 14400 3600 604800 86400

;; Query time: 112 msec
;; SERVER: 1.0.0.1#53(1.0.0.1)
;; WHEN: Tue Mar 19 03:24:49 Central Daylight Time 2024
;; MSG SIZE  rcvd: 87

toskium · Mar 19, 2024, 9:15 AM

Who is 10.0.0.1 ? (pfSense LAN ?)

@Gertjan: 1.0.0.1 is cloudflares nameserver.

On what device are you running this this command ? (a debian box ?)
On a pfSense LAN ?

@Gertjan: I ran the dig @1.0.0.1 command on both, pfSense and on a Debian host. Same results.

If "10.0.0.1" is pfSense, you could go here Services > DNS Resolver > Advanced Settings and set the Log Level to at least "3". Now, you can see the DNS requests for a SOA coming in, and see what unbound does with it.

@Gertjan: Just tried and tested. The request does not show up in the logs.

What are the DNS server(s) of this device ?

@Gertjan: the IP of my pfSense: 192.168.10.1

Not related , but : your are forwarding, you should disable DNSSEC.

@Gertjan: done.

@johnpoz, @Gertjan: regarding UDP it seems you are on to something here.

dig @1.0.0.1 SOA heise.de +tcp
dig @8.8.8.8 SOA heise.de +tcp

works flawlessly - only udp does not work as expected, tried on different nameservers as well.

To rule out firewall issues I currently disabled most rules, all that is active on LAN1 is
login-to-view

Floating rules are also currently disabled:
login-to-view

The machine I am testing from: 192.168.10.17 sits in the LAN1 network so in theory it should just be able to communicate.

johnpoz · Mar 19, 2024, 9:26 AM

@toskium so udp is not working to any outside dns?

I saw that your forwarding in unbound.. who are you forwarding too? Your isp dns? Maybe they block udp on 53 to anything other then their dns? or its possible that unbound while trying UDP and failing switches over to tcp.

I would sniff on your wan while you do dns queries to unbound on pfsense.. do you see it try udp when it forwards, and then do another query via tcp and get an answer? Unbound, or really and dns software where will/can do that - but dig is going to just try udp, unless you specific tell it to use tcp.

I don't see any rules listed there that would block you from talking to anything on UDP.

toskium · Mar 19, 2024, 9:37 AM

@johnpoz

I am currently forwarding to these (quad9)
login-to-view

dig was just a tool to help pinpointing the issue. The main usecase is traefiks dns challenge to request let's encrypt ssl certificates. Since my domain is using cloudflare nameservers it needs to be able to do a SOA lookup which it currently can't since its using udp.

johnpoz · Mar 19, 2024, 9:52 AM

@toskium if you can not use udp, you wouldn't be able to lookup anything via udp. Be it A, or MX or AAAA or NS or etc.. There is no possible way they are just blocking SOA via udp..

unbound and any dns server software normally will auto switch to tcp when udp fails.. I would again suggest you sniff on your wan.. Now ask unbound something via a client for something that is not cached already.. You should see unbound forward to where you forward via udp 53.. If that fails, it should fallback and try it via tcp..

toskium · Mar 19, 2024, 10:05 AM

@johnpoz
It seems to be somewhat related to dns handling of my pfSense dns resolver.

I did change my DNS servers on the pfSense from quad9 to cloudflare and now udp SOA lookups are working as expected but only to cloudflare.

So having these, instead of quad9:
login-to-view

Now allows me to do a:

dig @1.0.0.1 SOA heise.de

; <<>> DiG 9.18.24-1-Debian <<>> @1.0.0.1 SOA heise.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52452
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;heise.de.                      IN      SOA

;; ANSWER SECTION:
heise.de.               86400   IN      SOA     ns.heise.de. hostmaster.heise.de. 2024031918 14400 3600 604800 86400

;; Query time: 28 msec
;; SERVER: 1.0.0.1#53(1.0.0.1) (UDP)
;; WHEN: Tue Mar 19 11:01:23 CET 2024
;; MSG SIZE  rcvd: 87

But using:

dig @8.8.8.8 SOA heise.de

still fails.

@toskium if you can not use udp, you wouldn't be able to lookup anything via udp. Be it A, or MX or AAAA or NS or etc.. There is no possible way they are just blocking SOA via udp..

unbound and any dns server software normally will auto switch to tcp when udp fails.. I would again suggest you sniff on your wan.. Now ask unbound something via a client for something that is not cached already.. You should see unbound forward to where you forward via udp 53.. If that fails, it should fallback and try it via tcp..

Yes this is what I am wondering about as well. Since all other lookups just work as expected:
For instance:

dig @8.8.8.8 A heise.de

; <<>> DiG 9.18.24-1-Debian <<>> @8.8.8.8 A heise.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8245
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;heise.de.                      IN      A

;; ANSWER SECTION:
heise.de.               18260   IN      A       193.99.144.80

;; Query time: 16 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Tue Mar 19 11:04:02 CET 2024
;; MSG SIZE  rcvd: 53

johnpoz · Mar 19, 2024, 10:11 AM

@toskium what you have unbound do for forwarding has zero to do with you doing a directed query like

dig @1.0.0.1 SOA heise.de

Unless you were doing dns redirection.. I didn't see any rules in your firewall that were doing redirection, do you have a port forward setup for dns? Ie dns redirection? You wouldn't need a firewall rule specific if you had a any any rule.

What is in your port forward section of pfsense?

toskium · Mar 19, 2024, 10:16 AM

@johnpoz
Port forwarding is all empty except:
login-to-view

toskium · Mar 19, 2024, 11:54 AM

@johnpoz adding a dns redirect as a workaround helps for now. https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html

I just double checked on other pfSense hosts I am managing. On all of them the above dig@ command works without issue. The only real difference is that they are all single WAN.

I'd consider this somewhat solved for now, but I will have to investigate this behavior further, it seems I am missing something more or less obvious.

Anyways thanks for your assistance!