Certain domains failing in DNS Resolver/unbound
-
On my lab firewall, I'm running version 2.5.2, pfBlockerNG-devel 3.1.0_1. Everything works, except DNS lookups on a bunch of random domains that work via 8.8.8.8 or other external DNS servers.
I'm still working on a longer list of examples, but some domains I've noticed that always fail are t.co, ftx.us, and www.amazon.science. The t.co is a big one since it's used on Twitter links quite often.
[2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig t.co @localhost ; <<>> DiG 9.16.16 <<>> t.co @localhost ;; global options: +cmd ;; connection timed out; no servers could be reached [2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig t.co @8.8.8.8 ; <<>> DiG 9.16.16 <<>> t.co @8.8.8.8 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57575 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;t.co. IN A ;; ANSWER SECTION: t.co. 34 IN A 104.244.42.197 t.co. 34 IN A 104.244.42.69 t.co. 34 IN A 104.244.42.133 t.co. 34 IN A 104.244.42.5 ;; Query time: 15 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Wed May 18 11:04:27 EDT 2022 ;; MSG SIZE rcvd: 97The ones from that list above always say "connection timed out; no servers could be reached". The vast majority work fine:
[2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig reddit.com @localhost ; <<>> DiG 9.16.16 <<>> reddit.com @localhost ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44797 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;reddit.com. IN A ;; ANSWER SECTION: reddit.com. 21 IN A 151.101.65.140 reddit.com. 21 IN A 151.101.1.140 reddit.com. 21 IN A 151.101.193.140 reddit.com. 21 IN A 151.101.129.140 ;; Query time: 59 msec ;; SERVER: ::1#53(::1) ;; WHEN: Wed May 18 11:06:47 EDT 2022 ;; MSG SIZE rcvd: 103What options should I modify to fix this? Is there a way to a log file somewhere of DNS lookups that were tried and timed out? I do have "Enable Forwarding Mode" checked on the DNS Resolver page under "DNS Query Forwarding". If it's disabled, no DNS lookups work at all. Even with that enabled, Twitter's t.co and others still don't work
I need to fix this before I can safely move this FW into production
-
@clesports problem with resolving specific stuff normally is related to not being able to talk to server in the chain when resolving, either one of the gtld servers or even the specific authoritative servers.
I would do a dig +trace on pfsense to see where it might be failing.
example... I told it not to do dnssec, because that just clutters up the trace for such an example.
[22.01-RELEASE][admin@sg4860.local.lan]/root: dig t.co +trace +nodnssec ; <<>> DiG 9.16.23 <<>> t.co +trace +nodnssec ;; global options: +cmd . 4665 IN NS a.root-servers.net. . 4665 IN NS b.root-servers.net. . 4665 IN NS c.root-servers.net. . 4665 IN NS d.root-servers.net. . 4665 IN NS e.root-servers.net. . 4665 IN NS f.root-servers.net. . 4665 IN NS g.root-servers.net. . 4665 IN NS h.root-servers.net. . 4665 IN NS i.root-servers.net. . 4665 IN NS j.root-servers.net. . 4665 IN NS k.root-servers.net. . 4665 IN NS l.root-servers.net. . 4665 IN NS m.root-servers.net. ;; Received 239 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms co. 172800 IN NS ns1.cctld.co. co. 172800 IN NS ns3.cctld.co. co. 172800 IN NS ns2.cctld.co. co. 172800 IN NS ns4.cctld.co. co. 172800 IN NS ns5.cctld.co. co. 172800 IN NS ns6.cctld.co. ;; Received 441 bytes from 2001:500:12::d0d#53(g.root-servers.net) in 46 ms t.co. 3600 IN NS ns3.p34.dynect.net. t.co. 3600 IN NS d.r06.twtrdns.net. t.co. 3600 IN NS a.r06.twtrdns.net. t.co. 3600 IN NS b.r06.twtrdns.net. t.co. 3600 IN NS c.r06.twtrdns.net. t.co. 3600 IN NS d01-02.ns.twtrdns.net. t.co. 3600 IN NS d01-01.ns.twtrdns.net. t.co. 3600 IN NS ns4.p34.dynect.net. ;; Received 246 bytes from 2610:a1:1012::21#53(ns6.cctld.co) in 60 ms t.co. 30 IN A 104.244.42.5 t.co. 13999 IN NS a.r06.twtrdns.net. t.co. 13999 IN NS b.r06.twtrdns.net. t.co. 13999 IN NS c.r06.twtrdns.net. t.co. 13999 IN NS d.r06.twtrdns.net. t.co. 13999 IN NS d01-01.ns.twtrdns.net. t.co. 13999 IN NS d01-02.ns.twtrdns.net. t.co. 13999 IN NS ns1.p34.dynect.net. t.co. 13999 IN NS ns2.p34.dynect.net. t.co. 13999 IN NS ns3.p34.dynect.net. t.co. 13999 IN NS ns4.p34.dynect.net. ;; Received 256 bytes from 205.251.199.195#53(d.r06.twtrdns.net) in 12 ms [22.01-RELEASE][admin@sg4860.local.lan]/root: -
@johnpoz Thanks for your response. Here's the output from mine. It took a total of 65 seconds to run, is that normal?
[2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig t.co @localhost +trace +nodnssec ; <<>> DiG 9.16.16 <<>> t.co @localhost +trace +nodnssec ;; global options: +cmd . 36193 IN NS l.root-servers.net. . 36193 IN NS m.root-servers.net. . 36193 IN NS b.root-servers.net. . 36193 IN NS c.root-servers.net. . 36193 IN NS d.root-servers.net. . 36193 IN NS e.root-servers.net. . 36193 IN NS f.root-servers.net. . 36193 IN NS g.root-servers.net. . 36193 IN NS h.root-servers.net. . 36193 IN NS a.root-servers.net. . 36193 IN NS i.root-servers.net. . 36193 IN NS j.root-servers.net. . 36193 IN NS k.root-servers.net. ;; Received 239 bytes from ::1#53(localhost) in 219 ms co. 172800 IN NS ns5.cctld.co. co. 172800 IN NS ns4.cctld.co. co. 172800 IN NS ns3.cctld.co. co. 172800 IN NS ns1.cctld.co. co. 172800 IN NS ns2.cctld.co. co. 172800 IN NS ns6.cctld.co. ;; Received 441 bytes from 199.9.14.201#53(b.root-servers.net) in 67 ms t.co. 3600 IN NS ns4.p34.dynect.net. t.co. 3600 IN NS d01-01.ns.twtrdns.net. t.co. 3600 IN NS a.r06.twtrdns.net. t.co. 3600 IN NS ns3.p34.dynect.net. t.co. 3600 IN NS c.r06.twtrdns.net. t.co. 3600 IN NS d.r06.twtrdns.net. t.co. 3600 IN NS d01-02.ns.twtrdns.net. t.co. 3600 IN NS b.r06.twtrdns.net. ;; Received 218 bytes from 37.209.192.14#53(ns1.cctld.co) in 42 ms t.co. 1800 IN A 104.244.42.69 t.co. 1800 IN A 104.244.42.133 t.co. 1800 IN A 104.244.42.5 t.co. 1800 IN A 104.244.42.197 t.co. 13999 IN NS ns4.p34.dynect.net. t.co. 13999 IN NS ns1.p34.dynect.net. t.co. 13999 IN NS ns2.p34.dynect.net. t.co. 13999 IN NS a.r06.twtrdns.net. t.co. 13999 IN NS d.r06.twtrdns.net. t.co. 13999 IN NS b.r06.twtrdns.net. t.co. 13999 IN NS d01-02.ns.twtrdns.net. t.co. 13999 IN NS c.r06.twtrdns.net. t.co. 13999 IN NS ns3.p34.dynect.net. t.co. 13999 IN NS d01-01.ns.twtrdns.net. ;; Received 346 bytes from 108.59.163.34#53(ns3.p34.dynect.net) in 23 msAbout 1 minute of that time was waiting for Root Server B and the last one to respond
-
@clesports said in Certain domains failing in DNS Resolver/unbound:
;; Received 239 bytes from ::1#53(localhost) in 219 ms
this seems OFF
That should be like instant
See mine
;; Received 239 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
Keep in mind those are milliseconds not seconds ;) But even 0.2 seconds to get back roots from itself something is not right... That should be like 0 or 1ms.. 0.001 seconds..
edit: so you got back the gltd servers back from root in .067 seconds, but took you .219 seconds to get answer from yourself?? Something not right there
;; Received 441 bytes from 199.9.14.201#53(b.root-servers.net) in 67 ms
edit: is your unbound restarting? Do you have register dhcp leases set, that can cause unbound to restart a lot.. Which could present as long responses or failed responses.. if unbound is in the middle of a restart when you query.
-
@johnpoz said in Certain domains failing in DNS Resolver/unbound:
edit: is your unbound restarting? Do you have register dhcp leases set, that can cause unbound to restart a lot.. Which could present as long responses or failed responses.. if unbound is in the middle of a restart when you query.
I don't think so. I just restarted the box after your last message. In the DNS Resolver log file, I don't see any entry that has "restarting" in it. Just a few times I see stopping/starting which was probably me modifying pfBlocker settings trying to fix this issue.
Ran the commands below. The lookup for reddit.com was quick, the one for t.co still very slow
[2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig reddit.com @localhost +trace +nodnssec ; <<>> DiG 9.16.16 <<>> reddit.com @localhost +trace +nodnssec ;; global options: +cmd . 29952 IN NS f.root-servers.net. . 29952 IN NS g.root-servers.net. . 29952 IN NS h.root-servers.net. . 29952 IN NS i.root-servers.net. . 29952 IN NS j.root-servers.net. . 29952 IN NS k.root-servers.net. . 29952 IN NS l.root-servers.net. . 29952 IN NS m.root-servers.net. . 29952 IN NS a.root-servers.net. . 29952 IN NS b.root-servers.net. . 29952 IN NS c.root-servers.net. . 29952 IN NS d.root-servers.net. . 29952 IN NS e.root-servers.net. ;; Received 239 bytes from ::1#53(localhost) in 39 ms com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. ;; Received 835 bytes from 192.58.128.30#53(j.root-servers.net) in 40 ms reddit.com. 172800 IN NS ns-557.awsdns-05.net. reddit.com. 172800 IN NS ns-378.awsdns-47.com. reddit.com. 172800 IN NS ns-1029.awsdns-00.org. reddit.com. 172800 IN NS ns-1887.awsdns-43.co.uk. ;; Received 192 bytes from 192.48.79.30#53(j.gtld-servers.net) in 61 ms reddit.com. 300 IN A 151.101.1.140 reddit.com. 300 IN A 151.101.193.140 reddit.com. 300 IN A 151.101.65.140 reddit.com. 300 IN A 151.101.129.140 reddit.com. 172800 IN NS ns-1029.awsdns-00.org. reddit.com. 172800 IN NS ns-1887.awsdns-43.co.uk. reddit.com. 172800 IN NS ns-378.awsdns-47.com. reddit.com. 172800 IN NS ns-557.awsdns-05.net. ;; Received 240 bytes from 205.251.194.45#53(ns-557.awsdns-05.net) in 11 ms [2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig t.co @localhost +trace +nodnssec ; <<>> DiG 9.16.16 <<>> t.co @localhost +trace +nodnssec ;; global options: +cmd . 29935 IN NS i.root-servers.net. . 29935 IN NS j.root-servers.net. . 29935 IN NS k.root-servers.net. . 29935 IN NS l.root-servers.net. . 29935 IN NS m.root-servers.net. . 29935 IN NS a.root-servers.net. . 29935 IN NS b.root-servers.net. . 29935 IN NS c.root-servers.net. . 29935 IN NS d.root-servers.net. . 29935 IN NS e.root-servers.net. . 29935 IN NS f.root-servers.net. . 29935 IN NS g.root-servers.net. . 29935 IN NS h.root-servers.net. ;; Received 239 bytes from ::1#53(localhost) in 239 ms co. 172800 IN NS ns1.cctld.co. co. 172800 IN NS ns2.cctld.co. co. 172800 IN NS ns3.cctld.co. co. 172800 IN NS ns4.cctld.co. co. 172800 IN NS ns5.cctld.co. co. 172800 IN NS ns6.cctld.co. << it pauses a VERY long time before the next line >> ;; Received 411 bytes from 192.5.5.241#53(f.root-servers.net) in 11 ms t.co. 3600 IN NS ns4.p34.dynect.net. t.co. 3600 IN NS a.r06.twtrdns.net. t.co. 3600 IN NS d01-02.ns.twtrdns.net. t.co. 3600 IN NS ns3.p34.dynect.net. t.co. 3600 IN NS b.r06.twtrdns.net. t.co. 3600 IN NS c.r06.twtrdns.net. t.co. 3600 IN NS d.r06.twtrdns.net. t.co. 3600 IN NS d01-01.ns.twtrdns.net. ;; Received 218 bytes from 37.209.196.14#53(ns3.cctld.co) in 12 ms t.co. 30 IN A 104.244.42.197 t.co. 13999 IN NS a.r06.twtrdns.net. t.co. 13999 IN NS b.r06.twtrdns.net. t.co. 13999 IN NS c.r06.twtrdns.net. t.co. 13999 IN NS d.r06.twtrdns.net. t.co. 13999 IN NS d01-01.ns.twtrdns.net. t.co. 13999 IN NS d01-02.ns.twtrdns.net. t.co. 13999 IN NS ns1.p34.dynect.net. t.co. 13999 IN NS ns2.p34.dynect.net. t.co. 13999 IN NS ns3.p34.dynect.net. t.co. 13999 IN NS ns4.p34.dynect.net. ;; Received 256 bytes from 205.251.192.179#53(a.r06.twtrdns.net) in 12 msEdit: DHCP Registration is disabled in unbound
-
@clesports said in Certain domains failing in DNS Resolver/unbound:
;; Received 239 bytes from ::1#53(localhost) in 39 ms
this still seems high.. Why would a local response take so long? I mean its not really long .039 seconds.. But for asking yourself that seems like an age, it should be single ms for sure.. And normally faster than 1ms which is why it shows as 0..
Try doing a query to 127.0.0.1 vs localhost which seems to be calling out the ::1 IPv6 version of localhost.
But even with 219ms etc.. it resolved - its not like you actually timed out to anything, you were able to talk to all the ns.. So a client asking unbound for t.co should resolve.. those times are not anywhere so high that the client should say - ah he is not answering.. That is normally like 2 seconds for a client, before he gives up..
-
@clesports said in Certain domains failing in DNS Resolver/unbound:
co. 172800 IN NS ns6.cctld.co. << it pauses a VERY long time before the next line >>
;; Received 411 bytes from 192.5.5.241#53(f.root-servers.net) in 11 msbut it says it got response in 11ms or 0.011 seconds that is clearly not "long"
-
@johnpoz Yes, I'm not sure why it's saying it's 11ms when something else is holding it up for almost a minute from the end of "ns6.cctld.co" to the beginning of "Received 411 bytes".
When I changed my dig to 127.0.0.1 instead of it grabbing IPv6 from localhost, it did get the local response to 0 ms like we'd expect but still takes a minute to complete the full command.
I'll download a screen capture program and post a video when it's ready
-
@clesports hmmm - I wonder if its trying IPv6 and that is failing and then it fails over to doing IPv4
None of the responses are actually IPv6 in your trace - but you can see in mine, there is atleast 1 response that was via IPv6
; Received 441 bytes from 2001:500:12::d0d#53(g.root-servers.net) in 46 ms
Try turning off IPv6 in pfsense.. IPv6 can be very problematic when it thinks it should work, and it really isn't..
I would turn it off make sure no interfaces on pfsense actually have a IPv6 address, set your wan to none for IPv6..
How does it work now?
Can pfsense say ping a IPv6 address
[22.01-RELEASE][admin@sg4860.local.lan]/root: ping6 ipv6.google.com PING6(56=40+8+8 bytes) 2001:470:xxx:xxx::2 --> 2607:f8b0:4009:81b::200e 16 bytes from 2607:f8b0:4009:81b::200e, icmp_seq=0 hlim=118 time=52.111 ms 16 bytes from 2607:f8b0:4009:81b::200e, icmp_seq=1 hlim=118 time=48.045 ms -
@johnpoz I disabled IPv6 on all interfaces and the DHCP6 and RA server... still the same results. I even modified /etc/hosts and commended out the ipv6 line for localhost and that didn't help either.
I took a video capture running the same dig command so you can see what I see: https://www.transfernow.net/dl/dnsprob
There are been a few times after running it with the trace and nodnssec flags, the IP address resolves right away from a regular dig. After a few minutes, it's back to connection time out.
-
@johnpoz Interesting bit of info: If I disable DNSSEC Support, it works great. If I enable it again, back to the behavior in the video.
-
@clesports could be stalling on waiting for dnssec info.. Or the dnssec info is large, and your having an issue with UDP size..
I would sniff on your wan when you do that test.. So you can see traffic..
Could you do this test for me
[22.01-RELEASE][admin@sg4860.local.lan]/root: dig +short rs.dns-oarc.net TXT rst.x4090.rs.dns-oarc.net. rst.x4058.x4090.rs.dns-oarc.net. rst.x4064.x4090.rs.dns-oarc.net. [22.01-RELEASE][admin@sg4860.local.lan]/root:info here
https://www.dns-oarc.net/oarc/services/replysizetest
edit: also look for interception of your dns.. I went over how to do testing for this in this thread.
https://forum.netgate.com/post/1034563
edit2: Upon looking at your traces again - you do seem to be getting to really quick responses..
;; Received 218 bytes from 37.209.196.14#53(ns3.cctld.co) in 12 ms
That seems to be same for the different level of servers in the trace..
;; Received 411 bytes from 192.5.5.241#53(f.root-servers.net) in 11 ms
;; Received 256 bytes from 205.251.192.179#53(a.r06.twtrdns.net) in 12 msIts quite possible especially in light of your dnssec comment, that some shenanigans are going on... I would do some of the tests I pointed out in that thread I linked too.
edit3: You mentioned something about putting this into production.. Where exactly is pfsense currently in your network - is it directly connected to internet? Or is internal to your network, is it possible network pfsense is on could be redirecting dns?
-
@johnpoz Here's what I got from running that first command:
[2.5.2-RELEASE][root@pfSense1.home.arpa]/root: dig +short rs.dns-oarc.net TXT rst.x1363.rs.dns-oarc.net. rst.x1373.x1363.rs.dns-oarc.net. rst.x1344.x1373.x1363.rs.dns-oarc.net. "172.253.8.134 sent EDNS buffer size 1400" "172.253.8.134 DNS reply size limit is at least 1373"I think it's ok since my MTU is 1500 but I'm not sure. DNS is not being redirected because my dig @1.2.3.4 www.google.com command timed out (followed your linked forum post).
My production pfSense is behind an Internet load-balancer of sorts, but I would think shouldn't matter because these requests are UDP. The vast majority of traffic leaves 1 Internet leg. That prod. pfsense has DNS Resolver and Forwarder disabled.
Not sure if this is related, but when I ran the packet capture, I saw a bunch of packets to my DNS server like this (assigned via dhcp) even when I wasn't trying DNS lookups. Turned off pfblocker but they were still happening.
12:54:42.283726 IP (tos 0x0, ttl 64, id 30711, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.11885 > 10.1.0.185.53: [bad udp cksum 0x1519 -> 0xd84e!] 18342+% [1au] DNSKEY? co. ar: . OPT UDPsize=512 DO (31) 12:54:42.847699 IP (tos 0x0, ttl 64, id 17220, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.41917 > 10.1.0.185.53: [bad udp cksum 0x1519 -> 0x66db!] 17353+% [1au] DNSKEY? co. ar: . OPT UDPsize=512 DO (31) 12:54:43.222827 IP (tos 0x0, ttl 64, id 23923, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.15248 > 10.1.0.185.53: [bad udp cksum 0x1519 -> 0x7429!] 40616+% [1au] DNSKEY? co. ar: . OPT UDPsize=512 DO (31)On my external firewall, here was the capture output running the "dig t.co @127.0.0.1 +trace +nodnssec" command:
12:54:41.674000 IP (tos 0x0, ttl 64, id 46512, offset 0, flags [none], proto UDP (17), length 73) 10.1.0.38.9606 > 198.97.190.53.53: [udp sum ok] 8236 [1au] A? t.co. ar: . OPT UDPsize=4096 (45) 12:54:41.699642 IP (tos 0x0, ttl 51, id 29929, offset 0, flags [none], proto UDP (17), length 439) 198.97.190.53.53 > 10.1.0.38.9606: [udp sum ok] 8236- q: A? t.co. 0/6/13 ns: co. NS ns1.cctld.co., co. NS ns2.cctld.co., co. NS ns3.cctld.co., co. NS ns4.cctld.co., co. NS ns5.cctld.co., co. NS ns6.cctld.co. ar: ns1.cctld.co. A 37.209.192.14, ns2.cctld.co. A 37.209.194.14, ns3.cctld.co. A 37.209.196.14, ns4.cctld.co. A 156.154.103.25, ns5.cctld.co. A 156.154.104.25, ns6.cctld.co. A 156.154.105.25, ns1.cctld.co. AAAA 2001:dcd:1::14, ns2.cctld.co. AAAA 2001:dcd:2::14, ns3.cctld.co. AAAA 2001:dcd:3::14, ns4.cctld.co. AAAA 2610:a1:1010::21, ns5.cctld.co. AAAA 2610:a1:1011::21, ns6.cctld.co. AAAA 2610:a1:1012::21, . OPT UDPsize=1232 (411) 12:55:43.341876 IP (tos 0x0, ttl 64, id 25555, offset 0, flags [none], proto UDP (17), length 73) 10.1.0.38.7725 > 37.209.194.14.53: [udp sum ok] 2571 [1au] A? t.co. ar: . OPT UDPsize=4096 (45) 12:55:43.357935 IP (tos 0x0, ttl 51, id 23788, offset 0, flags [none], proto UDP (17), length 246) 37.209.194.14.53 > 10.1.0.38.7725: [udp sum ok] 2571- q: A? t.co. 0/8/1 ns: t.co. NS b.r06.twtrdns.net., t.co. NS a.r06.twtrdns.net., t.co. NS d.r06.twtrdns.net., t.co. NS d01-02.ns.twtrdns.net., t.co. NS ns4.p34.dynect.net., t.co. NS d01-01.ns.twtrdns.net., t.co. NS c.r06.twtrdns.net., t.co. NS ns3.p34.dynect.net. ar: . OPT UDPsize=1232 (218) 12:55:45.021631 IP (tos 0x0, ttl 64, id 40569, offset 0, flags [none], proto UDP (17), length 73) 10.1.0.38.30222 > 108.59.164.34.53: [udp sum ok] 27645 [1au] A? t.co. ar: . OPT UDPsize=4096 (45) 12:55:45.049542 IP (tos 0x0, ttl 51, id 63418, offset 0, flags [none], proto UDP (17), length 374) 108.59.164.34.53 > 10.1.0.38.30222: [udp sum ok] 27645*- q: A? t.co. 4/10/1 t.co. A 104.244.42.69, t.co. A 104.244.42.197, t.co. A 104.244.42.5, t.co. A 104.244.42.133 ns: t.co. NS d01-02.ns.twtrdns.net., t.co. NS a.r06.twtrdns.net., t.co. NS b.r06.twtrdns.net., t.co. NS d.r06.twtrdns.net., t.co. NS c.r06.twtrdns.net., t.co. NS ns1.p34.dynect.net., t.co. NS ns2.p34.dynect.net., t.co. NS ns3.p34.dynect.net., t.co. NS ns4.p34.dynect.net., t.co. NS d01-01.ns.twtrdns.net. ar: . OPT UDPsize=1232 (346) 12:56:00.383798 IP (tos 0x0, ttl 64, id 31475, offset 0, flags [none], proto UDP (17), length 76) 10.1.0.38.52598 > 202.12.27.33.53: [udp sum ok] 48489 [1au] A? nbc.com. ar: . OPT UDPsize=4096 (48)The large gap in time matches the previous video I posted.
-
@clesports said in Certain domains failing in DNS Resolver/unbound:
"172.253.8.134 DNS reply size limit is at least 1373"
Yeah that can be problematic - for large records, say dnssec.. Especially if you do not allow switching to tcp, or allow for tcp on 53.
You seem to have a mismatch going on, see here
OPT UDPsize=4096
But your not allowing for that size upstream of pfsense.. Be it your network, or your isp network.. But something upstream is limiting your size to 1373.
You could adjust this setting
And make sure your allowing for tcp 53 upstream so you can fallback to tcp when something is too large for your UDP limits.
-
@johnpoz I just checked the unbound logs and earlier today I was getting a bunch of messages like this
[42819:3] info: failed to prime trust anchor -- could not fetch DNSKEY rrset . DNSKEY INPlus I saw these in captures I was running from the lab firewall I've been testing with:
15:27:16.282653 IP (tos 0x0, ttl 128, id 9169, offset 0, flags [DF], proto UDP (17), length 60) 10.1.0.185.53 > 10.1.0.38.50941: [udp sum ok] 7159 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (32) 15:27:16.282658 IP (tos 0x0, ttl 128, id 9170, offset 0, flags [DF], proto UDP (17), length 60) 10.1.0.185.53 > 10.1.0.38.21142: [udp sum ok] 13473 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (32) 15:27:16.282705 IP (tos 0x0, ttl 64, id 16374, offset 0, flags [none], proto UDP (17), length 60) 10.1.0.38.15283 > 10.1.0.185.53: [bad udp cksum 0x151a -> 0x53af!] 11278+% [1au] DNSKEY? co. ar: . OPT UDPsize=4096 DO (32) 15:27:16.282706 IP (tos 0x0, ttl 64, id 25421, offset 0, flags [none], proto UDP (17), length 60) 10.1.0.38.56556 > 10.1.0.185.53: [bad udp cksum 0x151a -> 0xcf88!] 3835+% [1au] DNSKEY? co. ar: . OPT UDPsize=4096 DO (32) 15:27:16.282768 IP (tos 0x0, ttl 128, id 9171, offset 0, flags [DF], proto UDP (17), length 60) 10.1.0.185.53 > 10.1.0.38.24595: [udp sum ok] 52395 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (32) 15:27:16.282804 IP (tos 0x0, ttl 64, id 57620, offset 0, flags [none], proto UDP (17), length 60) 10.1.0.38.34577 > 10.1.0.185.53: [bad udp cksum 0x151a -> 0xbc52!] 30732+% [1au] DNSKEY? co. ar: . OPT UDPsize=4096 DO (32)So, with all the ServFail responses, could possibly be because of my internal DNS servers? Opened up TCP 53 on my external FW and it didn't seem to help, but I'm thinking this will all go away when I set a static IP and configure external DNS servers
Also, I modified the EDNS Buffer size to 4096 like you said, but still ran into the delays.
-
@clesports said in Certain domains failing in DNS Resolver/unbound:
I modified the EDNS Buffer size to 4096 like you said, but still ran into the delays.
That is not what I said - that is the default ;)
You could adjust this setting
I meant for you to lower to something below what your limited too.
You seem to be limited..
"172.253.8.134 sent EDNS buffer size 1400" "172.253.8.134 DNS reply size limit is at least 1373"So you need to adjust that to be inline with what you can do, which seems to be lower than 1400..
Try changing it to one of these
-
@johnpoz I'm sorry, I misunderstood what you were saying about the EDNS buffer size. I set it to 1232 and ran the reply size test and it returned the same information (buffer size 1400, size limit at least 1373)
The more I test, the more I'm starting to think the problem might be stemming from my Domain Controllers which are my DNS servers internally.
May 23 10:58:21 unbound 62587 [62587:2] info: generate keytag query _ta-4f66. NULL IN --Packet capture at that time-- 10:58:21.944462 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 79: (tos 0x0, ttl 64, id 4903, offset 0, flags [none], proto UDP (17), length 65) 10.1.0.38.5393 > 10.1.0.186.53: [bad udp cksum 0x1520 -> 0x6fb2!] 9469+ [1au] NULL? _ta-4f66. ar: . OPT UDPsize=1232 DO (37) 10:58:21.944791 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 79: (tos 0x0, ttl 128, id 17252, offset 0, flags [DF], proto UDP (17), length 65) 10.1.0.186.53 > 10.1.0.38.5393: [udp sum ok] 9469 ServFail q: NULL? _ta-4f66. 0/0/1 ar: . OPT UDPsize=4000 DO (37) 10:58:22.138338 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 79: (tos 0x0, ttl 64, id 6195, offset 0, flags [none], proto UDP (17), length 65) 10.1.0.38.13070 > 10.1.0.186.53: [bad udp cksum 0x1520 -> 0x8684!] 61469+% [1au] NULL? _ta-4f66. ar: . OPT UDPsize=1232 DO (37) 10:58:22.138686 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 79: (tos 0x0, ttl 128, id 17253, offset 0, flags [DF], proto UDP (17), length 65) 10.1.0.186.53 > 10.1.0.38.13070: [udp sum ok] 61469 ServFail q: NULL? _ta-4f66. 0/0/1 ar: . OPT UDPsize=4000 DO (37)The long delays/timeouts still happen, and during the 50-some-odd seconds of deadtime waiting for the 'dig trace' to complete, there are a TON of DNSKEY packets being sent that get ServFail responses 8-13 seconds later:
10:58:55.782487 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 3039, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.61614 > 10.1.0.186.53: [bad udp cksum 0x151a -> 0x9688!] 63270+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:55.891835 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 37474, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.47573 > 10.1.0.186.53: [bad udp cksum 0x151a -> 0x4032!] 33878+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:56.101717 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 50977, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.13093 > 10.1.0.186.53: [bad udp cksum 0x151a -> 0xcab4!] 32900+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:56.319455 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 2036, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.31293 > 10.1.0.186.53: [bad udp cksum 0x151a -> 0x5601!] 44575+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:56.729863 00:1a:8c:5c:ca:05 > d4:ae:52:a2:1d:3c, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 44968, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.42776 > 10.1.0.186.53: [bad udp cksum 0x151a -> 0xaa95!] 11440+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:57.079466 00:1a:8c:5c:ca:05 > c8:1f:66:bd:c7:61, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 19060, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.62855 > 10.1.0.185.53: [bad udp cksum 0x1519 -> 0xf5dd!] 37625+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:58:57.164326 00:1a:8c:5c:ca:05 > c8:1f:66:bd:c7:61, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 64, id 62098, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.38.36867 > 10.1.0.185.53: [bad udp cksum 0x1519 -> 0x6f74!] 32487+% [1au] DNSKEY? co. ar: . OPT UDPsize=1232 DO (31) 10:59:05.108434 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17326, offset 0, flags [DF], proto UDP (17), length 59) 10.1.0.186.53 > 10.1.0.38.47573: [udp sum ok] 33878 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:05.108503 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17325, offset 0, flags [DF], proto UDP (17), length 59) 10.1.0.186.53 > 10.1.0.38.61614: [udp sum ok] 63270 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:05.108504 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17327, offset 0, flags [DF], proto UDP (17), length 59) 10.1.0.186.53 > 10.1.0.38.13093: [udp sum ok] 32900 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:05.108562 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17328, offset 0, flags [DF], proto UDP (17), length 59) 10.1.0.186.53 > 10.1.0.38.31293: [udp sum ok] 44575 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:05.108563 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17329, offset 0, flags [DF], proto UDP (17), length 59) 10.1.0.186.53 > 10.1.0.38.42776: [udp sum ok] 11440 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:05.108564 d4:ae:52:a2:1d:3c > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 17330, offset 0, flags [DF], proto UDP (17), length 59) 10:59:08.475907 c8:1f:66:bd:c7:61 > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 11056, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.185.53 > 10.1.0.38.36867: [udp sum ok] 32487 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31) 10:59:08.475937 c8:1f:66:bd:c7:61 > 00:1a:8c:5c:ca:05, ethertype IPv4 (0x0800), length 73: (tos 0x0, ttl 128, id 11057, offset 0, flags [none], proto UDP (17), length 59) 10.1.0.185.53 > 10.1.0.38.62855: [udp sum ok] 37625 ServFail q: DNSKEY? co. 0/0/1 ar: . OPT UDPsize=4000 DO (31)Maybe there's a setting on the Domain Controllers to allow DNSSEC or something similar. No TCP session gets opened either, it sticks to UDP.
If I change my configured DNS servers in my lab FW to 1.1.1.1 or 8.8.8.8 and disabled the ones applied via DHCP, it seems to work (as of right now) and I see TCP DNS packets too for the DNSKEY part
-
@clesports you made no mention of pointing pfsense to any dns.. Out of the box pfsense RESOLVES.. the whole point of the +trace.
If your forwarding to something then no you shouldn't have dnssec checked. Where you forward either does dnssec for you, ie its a resolver or it doesn't.
If your not going to actually resolve with pfsense, then dnssec shouldn't be checked.
-
@johnpoz said in Certain domains failing in DNS Resolver/unbound:
@clesports you made no mention of pointing pfsense to any dns.. Out of the box pfsense RESOLVES.. the whole point of the +trace.
If your forwarding to something then no you shouldn't have dnssec checked. Where you forward either does dnssec for you, ie its a resolver or it doesn't.
If your not going to actually resolve with pfsense, then dnssec shouldn't be checked.
Ahh, that makes a big difference. In my initial post I did say I was doing DNS forwarding at the bottom of the post, but maybe it was missed:
I do have "Enable Forwarding Mode" checked on the DNS Resolver page under "DNS Query Forwarding". If it's disabled, no DNS lookups work at all. Even with that enabled, Twitter's t.co and others still don't work
I'll disable DNSSEC. I assume unbound in DNS Forwarding mode would still cache some results, and that's all that I'm looking to do since our Domain Controllers handle DNS records internally. I'm looking to point the DCs external requests to pfSense to take advantage of pfBlockerNG features.
Thank you for all your help John, I really do appreciate your knowledge and experience.
-
@clesports maybe I missed it - would of never had you do a trace if you forwarding. Forwarding is only helpful in troubleshooting a resolver.
If your forwarding - if you don't get an answer its the forwarder problem. or communication issue with the forwarder. What a trace shows is how resolver gets the answer.
dnssec is nothing but a problem if your forwarding - it shouldn't be checked. Either where you forward to is doing dnssec already, if not you asking for it doesn't do anything anyway other then wasted extra queries..
Makes sure if you have further questions with unbound, that you state in the beginning in forwarder mode - and where your actually forwarding too.. I looked over your first post again, and yeah you did say that. But its buried at the end after posts showing your attempt at lookup.
I assumed, my bad seems that Resolver/Unbound in the title that you were using default which is resolver..
If working for you now - all good, just took us longer to get to the problem.. Sorry about that.. if would of realized forwarding, prob could of gotten this straightened out in a couple of posts..
