Slow DNS after 22.05

johnpoz · Jul 11, 2022, 2:16 PM

@lohphat log might not catch the failure.. Depending on what the failure is - getting back a nx isn't a failure, etc.

What specific fqdn did not resolve - what did you get back from unbound, was it refused was it nx, was it a timeout?

Saying dns didn't work is like telling the mechanic your car is broke. Without details - have no clue to what is wrong or could be wrong.

You can for sure up the log level - also you can have it log queries and replies with

log-queries: yes
log-replies: yes

In the option box, etc.

But some actual details what you specific asked for, what was the response? And then doing a dig +trace is prob going to tell you more of what is failing then in the log..

lohphat · Jul 11, 2022, 2:55 PM

@johnpoz said in Slow DNS after 22.05:

@lohphat log might not catch the failure.. Depending on what the failure is - getting back a nx isn't a failure, etc.

What specific fqdn did not resolve - what did you get back from unbound, was it refused was it nx, was it a timeout?

That's the problem, I don't have a specific FQDN but I have cases where YT thumbnails don't load in a browser on my PC or iPad or on my Roku on my TV -- other similar sites which use distributed content like linkedin.com also fail at the same time. Since the content is embedded, getting the FQDN is more of a challenge.

You can for sure up the log level - also you can have it log queries and replies with

log-queries: yes
log-replies: yes

In the option box, etc.

Thanks! Will try this.

But some actual details what you specific asked for, what was the response? And then doing a dig +trace is prob going to tell you more of what is failing then in the log..

Once I can do that I will.

lohphat · Jul 11, 2022, 4:03 PM

@johnpoz

Here we go!

One second duckduckgo.com is resolved and the SAME second it's not.

Note: Not in forwarding mode.

Jul 11 12:00:23 	unbound 	85415 	[85415:0] info: 192.168.0.114 improving.duckduckgo.com. A IN SERVFAIL 0.015555 0 42
Jul 11 12:00:23 	unbound 	85415 	[85415:1] info: 192.168.0.114 duckduckgo.com. A IN SERVFAIL 0.010721 0 32
Jul 11 12:00:23 	unbound 	85415 	[85415:0] info: 192.168.0.114 external-content.duckduckgo.com. A IN SERVFAIL 0.011008 0 49
Jul 11 12:00:23 	unbound 	85415 	[85415:1] info: 192.168.0.114 improving.duckduckgo.com. AAAA IN NOERROR 0.000000 1 121
Jul 11 12:00:23 	unbound 	85415 	[85415:0] info: 192.168.0.114 external-content.duckduckgo.com. AAAA IN NOERROR 0.000000 1 114
Jul 11 12:00:23 	unbound 	85415 	[85415:1] info: 192.168.0.114 improving.duckduckgo.com. AAAA IN
Jul 11 12:00:23 	unbound 	85415 	[85415:1] info: 192.168.0.114 duckduckgo.com. AAAA IN NOERROR 0.000000 1 97
Jul 11 12:00:23 	unbound 	85415 	[85415:0] info: 192.168.0.114 external-content.duckduckgo.com. AAAA IN
Jul 11 12:00:23 	unbound 	85415 	[85415:0] info: 192.168.0.114 improving.duckduckgo.com. A IN
Jul 11 12:00:23 	unbound 	85415 	[85415:1] info: 192.168.0.114 duckduckgo.com. AAAA IN

lohphat · Jul 11, 2022, 4:09 PM

@johnpoz

Even more frustrating:

Jul 11 12:06:13 	unbound 	85415 	[85415:1] info: 192.168.0.114 duckduckgo.com. AAAA IN SERVFAIL 0.002623 0 32
Jul 11 12:06:13 	unbound 	85415 	[85415:0] info: 192.168.0.114 duckduckgo.com. A IN SERVFAIL 0.002413 0 32
Jul 11 12:06:13 	unbound 	85415 	[85415:1] info: 192.168.0.114 duckduckgo.com. AAAA IN
Jul 11 12:06:13 	unbound 	85415 	[85415:0] info: 192.168.0.114 duckduckgo.com. A IN

But when I do a DNS query in pfsense it resolves while my client is getting can't resolve errors. Presumably due to negative caching.

Results
Result 	Record type
52.149.246.39	A
Timings
Name server 	Query time
127.0.0.1	4 msec
9.9.9.9	15 msec
149.112.112.112	13 msec
2620:fe::fe	13 msec
2620:fe::9	14 msec

lohphat · Jul 11, 2022, 4:15 PM

@johnpoz

[22.05-RELEASE][root@pfSense.localdomain]/root: dig duckduckgo.com

; <<>> DiG 9.16.26 <<>> duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 19694
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;duckduckgo.com.                        IN      A

;; Query time: 5 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jul 11 12:12:35 EDT 2022
;; MSG SIZE  rcvd: 43

[22.05-RELEASE][root@pfSense.localdomain]/root: dig +trace duckduckgo.com

; <<>> DiG 9.16.26 <<>> +trace duckduckgo.com
;; global options: +cmd
.                       81871   IN      NS      c.root-servers.net.
.                       81871   IN      NS      g.root-servers.net.
.                       81871   IN      NS      a.root-servers.net.
.                       81871   IN      NS      d.root-servers.net.
.                       81871   IN      NS      i.root-servers.net.
.                       81871   IN      NS      l.root-servers.net.
.                       81871   IN      NS      m.root-servers.net.
.                       81871   IN      NS      e.root-servers.net.
.                       81871   IN      NS      b.root-servers.net.
.                       81871   IN      NS      h.root-servers.net.
.                       81871   IN      NS      f.root-servers.net.
.                       81871   IN      NS      k.root-servers.net.
.                       81871   IN      NS      j.root-servers.net.
.                       81871   IN      RRSIG   NS 8 0 518400 20220724050000 20220711040000 20826 . FZHYJQfCqUyzGmFm5rX40WBtbYNLRihMeE+ShEnhLJwV7XgAHidoNfh4 XAYL1BmOmBkMpZoffPF+BZ3/E4qQRPv9oMMd+uvEjAXoHMb6kI/2CMa4 ejTDy28Zr1YacUO70dk3bPF9TisNotaCzJFa1SOZ2a9WXDgzJe31DAyR rxchvGHz1/cLryangX9bgKWqu7MtE4TaXVpxYZM/L0eeUaxw1q15GXiG mM39zc+UHjA3TwjvUS7/7+6X9nvMbhDBNUUCxPMIKMMmoUUptBZ56HKU 1wQYf/iG+0XkZnH8hPVLZPoaj7EJQMbgpaGrHamDkprtmg7mRwRUPhgD jAA9kQ==
;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 3 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20220724050000 20220711040000 20826 . T1grSYw69rtfnUYr5D/ZqBqWYSDFbiIvUaPHYidsz6jvuEgEZUDKOmPN pjsHj7hW4HuLjfa5aAfnFS0oPuWvqiz8VfYjO4DIak/XiLOwwXhzQJBW iMRiKc++RUpEoHGZDDi8ZqpOKyM6prr6jST7N1Q01OjczwYF2iWKCRgV MA77kCdPpNbMZUuOaYlWqPwTt/DS0TUz9iaVdSR2mwNwNmxzhXyiDn0q erRTago7WSxUXWRi3yulg12Lz6vLQp4NAOD4Yj9T6jcdfVhk7j9RyKyG Awv6WhOLIv7LJheQ04jjK0dsWF1Jj6HigmU6C9I+qeBEPAKOksf1/is4 n5UsYg==
;; Received 1174 bytes from 2001:500:9f::42#53(l.root-servers.net) in 16 ms

duckduckgo.com.         172800  IN      NS      dns1.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns2.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns3.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns4.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      ns04.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns03.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns02.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns01.quack-dns.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20220718042400 20220711031400 37269 com. OqHMj8b2ZtnK/yLvlzh6HoFolzEZKjLjE2LBlMvL/9jNZsDvr9Mw1ssx HgckOUjba4gjVFDqpSKmyakdWtDsb0HaR13wvk7B2zphtOpqtY+R3G/6 IIjyMxwtxbdwCBF2isSyWisxdm0DVNfTtr6dDrB6xf7M5SDLXK+pweCy BwFoZ3vjutI7JQ19jEqjdpSL21ngvT/XV+tBJunhefJuOQ==
BN1F9K6MRAK5QR46K4F5GU46EBVE2B1H.com. 86400 IN NSEC3 1 1 0 - BN1G255EFBBQHJ10333PK7TVU3NBC6RJ NS DS RRSIG
BN1F9K6MRAK5QR46K4F5GU46EBVE2B1H.com. 86400 IN RRSIG NSEC3 8 2 86400 20220716050918 20220709035918 37269 com. tTFP0jB/KOhBDfPxWqmTqiR2bIzdS3j74Nj6YRAiBhoAmpCPSGkVXOq7 hxMHQhdBts2Qg9+pdzdcnhE1N4RG4hyXBQSqgYgqjK6eaceVsu27AKWh F0T0YWVR9xL6UFCYs9y2EeGe6vh1DjQ7gbDfBSJn0N+RPSx15pfaPw9R EVDRnh4AWbEKDFTRbdzd11eCUhhxVEAxy+Z0oUbl/wUkvQ==
;; Received 943 bytes from 192.52.178.30#53(k.gtld-servers.net) in 85 ms

duckduckgo.com.         200     IN      A       52.149.246.39
;; Received 59 bytes from 198.51.45.5#53(dns2.p05.nsone.net) in 17 ms

johnpoz · Jul 11, 2022, 4:17 PM

@lohphat you understand A and AAAA are not the same thing right.

So you need to figureout why your getting servfail. for duckduckgo.com

What does the trace look like?


[22.05-RELEASE][admin@sg4860.local.lan]/: dig duckduckgo.com +trace

; <<>> DiG 9.16.26 <<>> duckduckgo.com +trace
;; global options: +cmd
.                       48564   IN      NS      f.root-servers.net.
.                       48564   IN      NS      m.root-servers.net.
.                       48564   IN      NS      a.root-servers.net.
.                       48564   IN      NS      g.root-servers.net.
.                       48564   IN      NS      d.root-servers.net.
.                       48564   IN      NS      i.root-servers.net.
.                       48564   IN      NS      l.root-servers.net.
.                       48564   IN      NS      c.root-servers.net.
.                       48564   IN      NS      e.root-servers.net.
.                       48564   IN      NS      j.root-servers.net.
.                       48564   IN      NS      k.root-servers.net.
.                       48564   IN      NS      h.root-servers.net.
.                       48564   IN      NS      b.root-servers.net.
.                       48564   IN      RRSIG   NS 8 0 518400 20220723170000 20220710160000 20826 . cn2zeGJHs3h6a1nWmMh7VUYzs3qfSO3h2F/J38SVG7SCpOYKwgiAiOIA 0n4zaeM/AE4T4Upj+gjHkKt6FO6Ksv3bbe+iDVjiiMLlEdGOJPoR6fi8 4Pi48jKsYhIXRirYwmvNH23aRM3J+HLdKFUwCRYp71G4o72yt5SFg0xZ HCteNbA4qOfBERRA05LzofsRqW1dexcylxbtZQZVxCzQ4TdYWsq1+Wl+ AfVQrUEoL7L14wQrUlVqu980qbKO1wQBK8idnRQv548u3O4su+OyQ+Hc 1VjhrRt8BG84JgVgURADoiGeVs/Ojn/tOXIlhUmuNIZdDGR7BT8MYl1Z tUXTKw==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20220724050000 20220711040000 20826 . T1grSYw69rtfnUYr5D/ZqBqWYSDFbiIvUaPHYidsz6jvuEgEZUDKOmPN pjsHj7hW4HuLjfa5aAfnFS0oPuWvqiz8VfYjO4DIak/XiLOwwXhzQJBW iMRiKc++RUpEoHGZDDi8ZqpOKyM6prr6jST7N1Q01OjczwYF2iWKCRgV MA77kCdPpNbMZUuOaYlWqPwTt/DS0TUz9iaVdSR2mwNwNmxzhXyiDn0q erRTago7WSxUXWRi3yulg12Lz6vLQp4NAOD4Yj9T6jcdfVhk7j9RyKyG Awv6WhOLIv7LJheQ04jjK0dsWF1Jj6HigmU6C9I+qeBEPAKOksf1/is4 n5UsYg==
;; Received 1174 bytes from 2001:503:c27::2:30#53(j.root-servers.net) in 120 ms

duckduckgo.com.         172800  IN      NS      dns1.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns2.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns3.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      dns4.p05.nsone.net.
duckduckgo.com.         172800  IN      NS      ns04.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns03.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns02.quack-dns.com.
duckduckgo.com.         172800  IN      NS      ns01.quack-dns.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20220718042400 20220711031400 37269 com. OqHMj8b2ZtnK/yLvlzh6HoFolzEZKjLjE2LBlMvL/9jNZsDvr9Mw1ssx HgckOUjba4gjVFDqpSKmyakdWtDsb0HaR13wvk7B2zphtOpqtY+R3G/6 IIjyMxwtxbdwCBF2isSyWisxdm0DVNfTtr6dDrB6xf7M5SDLXK+pweCy BwFoZ3vjutI7JQ19jEqjdpSL21ngvT/XV+tBJunhefJuOQ==
BN1F9K6MRAK5QR46K4F5GU46EBVE2B1H.com. 86400 IN NSEC3 1 1 0 - BN1G255EFBBQHJ10333PK7TVU3NBC6RJ NS DS RRSIG
BN1F9K6MRAK5QR46K4F5GU46EBVE2B1H.com. 86400 IN RRSIG NSEC3 8 2 86400 20220716050918 20220709035918 37269 com. tTFP0jB/KOhBDfPxWqmTqiR2bIzdS3j74Nj6YRAiBhoAmpCPSGkVXOq7 hxMHQhdBts2Qg9+pdzdcnhE1N4RG4hyXBQSqgYgqjK6eaceVsu27AKWh F0T0YWVR9xL6UFCYs9y2EeGe6vh1DjQ7gbDfBSJn0N+RPSx15pfaPw9R EVDRnh4AWbEKDFTRbdzd11eCUhhxVEAxy+Z0oUbl/wUkvQ==
;; Received 943 bytes from 2001:502:1ca1::30#53(e.gtld-servers.net) in 35 ms

duckduckgo.com.         200     IN      A       40.89.244.232
;; Received 59 bytes from 2a00:edc0:6259:7:5::2#53(dns2.p05.nsone.net) in 71 ms

edit: so trace works but asking unbound fails.. with servfail.. But other stuff is all working?

lohphat · Jul 11, 2022, 4:18 PM

@johnpoz Yes I know the difference between A and AAAA.

The dig trace is included in last message.

lohphat · Jul 11, 2022, 4:25 PM

@johnpoz said in Slow DNS after 22.05:

edit: so trace works but asking unbound fails.. with servfail.. But other stuff is all working?

e.g. YT is now failing but only on some FQDNs -- enough to break the UI and prevent thumbnails from loading.

Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 yt3.ggpht.com. AAAA IN SERVFAIL 0.146761 0 31
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 photos-ugc.l.googleusercontent.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 yt3.ggpht.com. AAAA IN SERVFAIL 0.114413 0 31
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN NOERROR 0.000000 1 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN NOERROR 0.000000 1 72
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. A IN NOERROR 0.000000 1 60
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN NOERROR 0.000000 1 60
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN NOERROR 0.032962 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN NOERROR 0.026360 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN NOERROR 0.058236 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN NOERROR 0.019413 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN NOERROR 0.051366 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN NOERROR 0.000000 1 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. A IN NOERROR 0.000000 1 60
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN NOERROR 0.037295 0 72
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN NOERROR 0.000000 1 60
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN NOERROR 0.026594 0 60
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn0.gstatic.com. A IN NOERROR 0.020316 0 60
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn3.gstatic.com. A IN NOERROR 0.021593 0 60
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn0.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 encrypted-tbn0.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 yt3.ggpht.com. A IN NOERROR 0.058754 0 92
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn3.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 yt3.ggpht.com. A IN NOERROR 0.026181 0 92
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn3.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 encrypted-tbn2.gstatic.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 i.ytimg.com. A IN SERVFAIL 0.004822 0 29
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 i.ytimg.com. AAAA IN SERVFAIL 0.002218 0 29
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 i.ytimg.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 i.ytimg.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 yt3.ggpht.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 yt3.ggpht.com. A IN
Jul 11 12:21:18 	unbound 	85415 	[85415:0] info: 192.168.0.114 yt3.ggpht.com. AAAA IN
Jul 11 12:21:18 	unbound 	85415 	[85415:1] info: 192.168.0.114 yt3.ggpht.com. A IN

johnpoz · Jul 11, 2022, 5:19 PM

@lohphat said in Slow DNS after 22.05:

i.ytimg.com

So figure out why servfail - you sure your dns not being intercepted and you failing dnssec?

I am having zero issues resolving that, and your trace for example looks fine.

I would sniff to find out why our having issues.. So you query that directly and you get back servfail - what does unbound show for its cache that - ask it how it would look it up, etc..

[22.05-RELEASE][admin@sg4860.local.lan]/: unbound-control -c /var/unbound/unbound.conf lookup i.ytimg.com
The following name servers are used for lookup of i.ytimg.com.
;rrset 25504 4 0 2 0
ytimg.com.      25504   IN      NS      ns2.google.com.
ytimg.com.      25504   IN      NS      ns1.google.com.
ytimg.com.      25504   IN      NS      ns3.google.com.
ytimg.com.      25504   IN      NS      ns4.google.com.
;rrset 44984 1 0 1 0
ns4.google.com. 44984   IN      A       216.239.38.10
;rrset 44984 1 0 1 0
ns4.google.com. 44984   IN      AAAA    2001:4860:4802:38::a
;rrset 44984 1 0 1 0
ns3.google.com. 44984   IN      A       216.239.36.10
;rrset 44984 1 0 1 0
ns3.google.com. 44984   IN      AAAA    2001:4860:4802:36::a
;rrset 44984 1 0 1 0
ns1.google.com. 44984   IN      A       216.239.32.10
;rrset 44984 1 0 1 0
ns1.google.com. 44984   IN      AAAA    2001:4860:4802:32::a
;rrset 44984 1 0 1 0
ns2.google.com. 44984   IN      A       216.239.34.10
;rrset 44984 1 0 1 0
ns2.google.com. 44984   IN      AAAA    2001:4860:4802:34::a
Delegation with 4 names, of which 0 can be examined to query further addresses.
It provides 8 IP addresses.
2001:4860:4802:34::a    not in infra cache.
216.239.34.10           expired, rto 3453376 msec, tA 0 tAAAA 0 tother 0.
2001:4860:4802:32::a    not in infra cache.
216.239.32.10           expired, rto 3453376 msec, tA 0 tAAAA 0 tother 0.
2001:4860:4802:36::a    not in infra cache.
216.239.36.10           expired, rto 3453376 msec, tA 0 tAAAA 0 tother 0.
2001:4860:4802:38::a    not in infra cache.
216.239.38.10           rto 242 msec, ttl 304, ping 10 var 58 rtt 242, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
[22.05-RELEASE][admin@sg4860.local.lan]/:

lohphat · Jul 11, 2022, 5:50 PM

@johnpoz

Well, this failure mode is intermittent -- it hits then resolves, then his again later, rinse, repeat. It doesn't happen in DNS Forward mode. So I'm guessing whatever it is, is happening in the local cache.

The failure mode affects different devices on different internal networks which are of different architectures: Win11, iPad OS, Roku, Android.

The behavior started after the 22.05 update almost immediately. No other changes other than the base image were made.

The following advanced options are set (and were set in 22.01):

Prefetch DNS Key Support
Harden DNSSEC Data
Keep Probing
Experimental Bit 0x20 Support

When things start failing again, I'll post the deeper query results.

lohphat · Jul 11, 2022, 6:18 PM

@johnpoz

Here you go:

[22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup spectrum.s3.amazonaws.com
The following name servers are used for lookup of spectrum.s3.amazonaws.com.
;rrset 5502 4 0 7 0
s3.amazonaws.com.       5502    IN      NS      ns-1300.awsdns-34.org.
s3.amazonaws.com.       5502    IN      NS      ns-1579.awsdns-05.co.uk.
s3.amazonaws.com.       5502    IN      NS      ns-63.awsdns-07.com.
s3.amazonaws.com.       5502    IN      NS      ns-771.awsdns-32.net.
;rrset 77561 1 0 8 0
ns-771.awsdns-32.net.   77561   IN      A       205.251.195.3
;rrset 77561 1 0 8 0
ns-771.awsdns-32.net.   77561   IN      AAAA    2600:9000:5303:300::1
;rrset 77561 1 0 8 0
ns-63.awsdns-07.com.    77561   IN      A       205.251.192.63
;rrset 77650 1 0 8 0
ns-63.awsdns-07.com.    77650   IN      AAAA    2600:9000:5300:3f00::1
;rrset 77561 1 0 8 0
ns-1579.awsdns-05.co.uk.        77561   IN      A       205.251.198.43
;rrset 77561 1 0 8 0
ns-1579.awsdns-05.co.uk.        77561   IN      AAAA    2600:9000:5306:2b00::1
;rrset 77561 1 0 8 0
ns-1300.awsdns-34.org.  77561   IN      A       205.251.197.20
;rrset 77561 1 0 8 0
ns-1300.awsdns-34.org.  77561   IN      AAAA    2600:9000:5305:1400::1
Delegation with 4 names, of which 0 can be examined to query further addresses.
It provides 8 IP addresses.
2600:9000:5305:1400::1  expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
205.251.197.20          expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
2600:9000:5306:2b00::1  expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
205.251.198.43          expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
2600:9000:5300:3f00::1  not in infra cache.
205.251.192.63          expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
2600:9000:5303:300::1   expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0.
205.251.195.3           not in infra cache.
[22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup lycraservice-pa.googleapis.com
The following name servers are used for lookup of lycraservice-pa.googleapis.com.
;rrset 74731 4 0 2 0
googleapis.com. 74731   IN      NS      ns2.google.com.
googleapis.com. 74731   IN      NS      ns1.google.com.
googleapis.com. 74731   IN      NS      ns3.google.com.
googleapis.com. 74731   IN      NS      ns4.google.com.
;rrset 74608 1 0 1 0
ns4.google.com. 74608   IN      A       216.239.38.10
;rrset 74608 1 0 1 0
ns4.google.com. 74608   IN      AAAA    2001:4860:4802:38::a
;rrset 74608 1 0 1 0
ns3.google.com. 74608   IN      A       216.239.36.10
;rrset 74608 1 0 1 0
ns3.google.com. 74608   IN      AAAA    2001:4860:4802:36::a
;rrset 74608 1 0 1 0
ns1.google.com. 74608   IN      A       216.239.32.10
;rrset 74608 1 0 1 0
ns1.google.com. 74608   IN      AAAA    2001:4860:4802:32::a
;rrset 74608 1 0 1 0
ns2.google.com. 74608   IN      A       216.239.34.10
;rrset 74608 1 0 1 0
ns2.google.com. 74608   IN      AAAA    2001:4860:4802:34::a
Delegation with 4 names, of which 0 can be examined to query further addresses.
It provides 8 IP addresses.
2001:4860:4802:34::a    expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0.
216.239.34.10           expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0.
2001:4860:4802:32::a    rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.32.10           not in infra cache.
2001:4860:4802:36::a    rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.36.10           not in infra cache.
2001:4860:4802:38::a    rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.38.10           not in infra cache.
[22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup streetviewpixels-pa.googleapis.com
The following name servers are used for lookup of streetviewpixels-pa.googleapis.com.
;rrset 74647 4 0 2 0
googleapis.com. 74647   IN      NS      ns2.google.com.
googleapis.com. 74647   IN      NS      ns1.google.com.
googleapis.com. 74647   IN      NS      ns3.google.com.
googleapis.com. 74647   IN      NS      ns4.google.com.
;rrset 74524 1 0 1 0
ns4.google.com. 74524   IN      A       216.239.38.10
;rrset 74524 1 0 1 0
ns4.google.com. 74524   IN      AAAA    2001:4860:4802:38::a
;rrset 74524 1 0 1 0
ns3.google.com. 74524   IN      A       216.239.36.10
;rrset 74524 1 0 1 0
ns3.google.com. 74524   IN      AAAA    2001:4860:4802:36::a
;rrset 74524 1 0 1 0
ns1.google.com. 74524   IN      A       216.239.32.10
;rrset 74524 1 0 1 0
ns1.google.com. 74524   IN      AAAA    2001:4860:4802:32::a
;rrset 74524 1 0 1 0
ns2.google.com. 74524   IN      A       216.239.34.10
;rrset 74524 1 0 1 0
ns2.google.com. 74524   IN      AAAA    2001:4860:4802:34::a
Delegation with 4 names, of which 0 can be examined to query further addresses.
It provides 8 IP addresses.
2001:4860:4802:34::a    expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0.
216.239.34.10           expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0.
2001:4860:4802:32::a    rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.32.10           not in infra cache.
2001:4860:4802:36::a    rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.36.10           not in infra cache.
2001:4860:4802:38::a    rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
216.239.38.10           not in infra cache.
[22.05-RELEASE][root@pfSense.localdomain]/root:

chrcoluk · Jul 12, 2022, 5:37 AM

@lohphat try disabling the experimental bit 0x20 support, and also try enabling the "serve expired" option which helps a ton with cdn dns performance.

Gertjan · Jul 12, 2022, 8:02 AM

@lohphat said in Slow DNS after 22.05:

My new question is why the alias being resolved used while the test rule I use the alias in, isn't being used at all.

??

The Alias is defined.
So it get resolved regularly.
An alias can be used not only in a firewall rule, but also some where else.
Or even not used at all : as long as it is defined, it get resolved.

lohphat · Jul 12, 2022, 2:34 PM

@johnpoz said in Slow DNS after 22.05:

@lohphat said in Slow DNS after 22.05:

DNS forwarding and DNSSEC enabled:

That is never a good idea, if your going to forward where you forward either does dnssec or it doesn't you asking for it does nothing - other than problematic.

Interesting side-effect if DNSSEC is disabled, I get a warning if I try to change the Advanced Settings:

The following input errors were detected:

    Harden DNSSEC Data option can only be enabled if DNSSEC support is enabled.

Another example of where info text could be clearer when there are inter-dependencies of settings. I want DNSSEC, but it's not needed if you're forwarding, but then the setting on another page is dependent on it. I didn't get a warning that it was set when I disabled DNSSEC on the main page. Confusing.

Also: The setting default should be included in each setting text so that as you're tweaking over time, knowing the default setting is helpful to return settings to original.

lohphat · Jul 12, 2022, 11:05 PM

@gertjan But that behavior is not obvious.

Getting DNS lookup failures in the log took me a while to trace where I had those references. I had long forgotten the alias I had created to test a f/w rule for DNS blocking.

Gertjan · Jul 13, 2022, 8:44 AM

@lohphat said in Slow DNS after 22.05:

Confusing

True.
But changing a setting on a first page that automatically change (disables) a setting on a second page is more confusing.
All this is IMHO of course.

Right now, you are hinted to disable first the extra 'special' DNSSEC settings and the second page, and then, at last; disable DNSSEC all together on the main page.

And strange .... DNSSEC works so well for me the last several years already.
I'm using DNSSEC, as pfSense is setup out of the box to use it.
Because a good 'flat' classic Internet connection should not intervene with my outgoing traffic.
If this wasn't the case, I would change ISP ASAP.

These :

login-to-view

I've checked years ago.
Never had the need to remove them (which means I rarely visit sites with DNSSEC issues, I guess)

lohphat · Jul 13, 2022, 10:32 AM

@gertjan After all the testing the setting which has seemed to solve my problem is to enable "Serve Expired".

That was not enabled when I was running pre 22.05 so I wonder what changed between 22.01 and 22.05 which changed the behavior in my environment.

So far, after re-enabling DNSSEC and the Experimental 0x 20 support, things are working again -- just "Serve Expired" seems to have been the issue.

johnpoz · Jul 13, 2022, 11:05 AM

@lohphat I would turn on prefetch as well.

You also might want to set a min ttl. You mention seems to be problem with cdn stuff - possible the ttl is so freaking small that if your having time problems with resolving - you could run into timeouts..

I looked at one example you gave - 300 seconds..

;; QUESTION SECTION:
;i.ytimg.com.                   IN      A

;; ANSWER SECTION:
i.ytimg.com.            300     IN      A       142.250.191.150
i.ytimg.com.            300     IN      A       142.250.191.182
i.ytimg.com.            300     IN      A       142.250.191.214
i.ytimg.com.            300     IN      A       142.250.191.246
i.ytimg.com.            300     IN      A       142.251.32.22
i.ytimg.com.            300     IN      A       142.250.190.22
i.ytimg.com.            300     IN      A       142.250.190.54
i.ytimg.com.            300     IN      A       142.250.190.86
i.ytimg.com.            300     IN      A       142.250.190.118
i.ytimg.com.            300     IN      A       142.250.190.150
i.ytimg.com.            300     IN      A       172.217.0.182
i.ytimg.com.            300     IN      A       172.217.1.118
i.ytimg.com.            300     IN      A       172.217.2.54
i.ytimg.com.            300     IN      A       172.217.4.54
i.ytimg.com.            300     IN      A       172.217.5.22
i.ytimg.com.            300     IN      A       172.217.4.86

istacey · Jul 13, 2022, 8:57 PM

Hi, +1 here on the issue with erratic behaviour on DNS lookups since 22.05 update.

I'm going to read the full chain here, but lots of similarities in the bits I've skim-read.

Lots of these entries in the logs for unbound

Jul 13 21:43:18 unbound 982 [982:1] error: recvfrom 22 failed: Protocol not available

I'm using Cloudflare DNS servers, not allowing my WAN connections DHCP settings to flow through and have things set to "use remote, ignore local". I don't have these DNS servers set in my DHCP settings.

DNS Forwarder is inactive, DNS Resolver is active.

lohphat · Jul 15, 2022, 2:41 PM

@istacey

Over the last few days the only change I've made in addition to "Serve Expired" and add a minimum TTL of 900 sec (setting the help text doesn't specify units, but I have a long-standing complaint on the lack of min detail in setting help text). I also turned off "Use Experimental 0x 20" for DNS spoofing; this too over several days proved unstable (and a change between 22.01 to 22.05 as it was working fine before).

So yes, something has significantly changed in unbound in the last release.