23.01 breaks DNS resolver and pFblocker

[lpfw]

Nothing but problems since the upgrade with resolving DNS via the firewall.
Disabling pfblocker doesn't help either. DNS just randomly won't resolve after the upgrade. So mad at myself for upgrading.

Nothing useful in the logs or status of DNS resolver.

[digs]

@lpfw same

[stephenw10]

The resolution failures are intermittent? Random domains?

Do you have IPv6 configured?

[Draco]

I am seeing this same problem, though in my case the DNS resolution is taking a long time for non-local DNS names that are not (or no longer) in the cache. Once in the cache, things seem fine (until the entry expires). I do not have Forwarding on and have tried with DNSSEC on and off. This is causing failures on apps that try to update overnight and my overnight backups.

I am going to repave back to 22.05. Glad I always keep a USB "repave" stick before upgrades!

[Gertjan]

@draco said in 23.01 breaks DNS resolver and pFblocker:

non-local DNS names that are not (or no longer) in the cache

Stop-gap solution :

From now on, host names that expire will get updated - so they'll be fresh and available when needed;

and them, instead of using the word "problems", let's go hunt down why your resolver has issues reaching root servers (any of them ?) - top level domain servers (all of them ?) and domain name servers.

What is your WAN ? Is it special ? VPN ? A special ISP ?
VPN ? IPv4 works fine ? Half broken IPv6 ?

[johnpoz]

@gertjan good settings, another one I might suggest is serve zero. I have been using it for years, and have never had a problem with it.

serve 0 allows unbound to serve up the last IP it had in cache for some fqdn even if the ttl has expired. The ttl it hands off to the client will be 0, so if for some reason that doesn't work the client will have to ask again for that fqdn, and by this time unbound will have looked it up.

[Draco]

@gertjan My WAN is CLink DSL (modem bridged to pfSense which handles the PPPoE login to avoid) with Starlink as a failover (DHCP connection). The failover gateway has not been flipping back and forth when I have these problems.

IPv4 only on both WANs. Connectivity is solid. It’s just the laggy DNS lookups that are killing me. It causes my online backups to time out and fail, websites either fail to load entirely, or take forever as they (my guess) wait for multiple DNS responses from Unbound for various “assets” on the page (FlickR is one example of this).

The only thing that changed in my network config is my upgrade from 22.05 to 23.01 on my SG-5100, insofar as I know.

[johnpoz]

@draco said in 23.01 breaks DNS resolver and pFblocker:

wait for multiple DNS responses from Unbound for various “assets” on the page (FlickR is one example of this).

You understand once unbound caches the NS for say the gtld servers (.com, .net, .etc) it doesn't have to look them from roots every time. So if you ask for somedomain.com it knows the gtld to talk to get the authoritative ns for somedomain.com

Once it has the NS for somedomain.com and you want to look up somehost.somedomain.com it can just ask the NS directly.

The full resolve for somedomain.com should only take at most a few hundred ms anyway on the first time around. This would and shouldn't cause you any sort of noticeable delay in loading websites.

If you have some problem resolving - I would look to where your having slowness.. A dig +trace to what your looking up will give you the response times and what and how many NS are in the path for a full resolve from roots. Or if you having issues talking to any specific NS in the path of a full resolve, etc.

example.

23.01-RELEASE][admin@sg4860.local.lan]/: dig www.reditt.com +trace

; <<>> DiG 9.18.8 <<>> www.reditt.com +trace
;; global options: +cmd
.                       76524   IN      NS      h.root-servers.net.
.                       76524   IN      NS      i.root-servers.net.
.                       76524   IN      NS      j.root-servers.net.
.                       76524   IN      NS      k.root-servers.net.
.                       76524   IN      NS      l.root-servers.net.
.                       76524   IN      NS      m.root-servers.net.
.                       76524   IN      NS      a.root-servers.net.
.                       76524   IN      NS      b.root-servers.net.
.                       76524   IN      NS      c.root-servers.net.
.                       76524   IN      NS      d.root-servers.net.
.                       76524   IN      NS      e.root-servers.net.
.                       76524   IN      NS      f.root-servers.net.
.                       76524   IN      NS      g.root-servers.net.
.                       76524   IN      RRSIG   NS 8 0 518400 20230317050000 20230304040000 951 . kJaY9uAyEtbTnrjZ1qAQTsqHExUgSViSqmXstFQXUmBOgAbAHKQlp9Nj BCAb0pUbm3sDWOrGvOaqxN6QKFXd8331v6lxtsDKd3kIGE5Wo7kLwzw4 XzZeGRwfuRPnmwXtfYnJTo+X4tGgg2xK6c0uy5QdsFVzHEPwJNXURZVE rXQ/erzXJmKUXFuZim8sfm7UjTTJJsBwk8+P8uM+B9CKDtfE0CvxtyIS BbGi9pg4PDlJz0zB3V9VM/9+IcJuQ4NfnBDvw3pD9Q0LVx9qN2GzG1TK 06r6LMEBB9RRhO5wkZ7UwZuVzloYxntIpBMVL3zdTl2vVCIQFlzqSqJL 5OfW1A==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

com.                    172800  IN      NS      m.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20230317050000 20230304040000 951 . L19NBeJKQI2xghmOhCM0yzP7QhObOVesyKtpb5hv9pyEhGocrShcO5uR lQnvOIUHTVhSKlCKraGu0NH844iKR9g795Q8HuQKPiAx8RlnHcdY1dBE ljnZAk7jZmPbsBKafKEleyWcE2rhBfrMSAEPu8p7TLgrpI3AEA4lisUB uQAU7Sj0KzphiBSP3UqJxx48r1u4NNxqki8WXRO8zUhPkanruAbDpVcx VcmemRMmi9pDeJYU35nPxR7h86vWJ1av0AjAZ+LXZV70GsP/h9/540fU cbfRmJnPFyKFBxPZrX5lAUwW67K7G8zCFJKufDvUYbvXJg2F80d9Zo20 Y+4dqA==
;; Received 1205 bytes from 192.112.36.4#53(g.root-servers.net) in 24 ms

reditt.com.             172800  IN      NS      421.ns1.above.com.
reditt.com.             172800  IN      NS      421.ns2.above.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20230311052252 20230304041252 36739 com. exNISCQI4v/S0m9ksCZH3zghILb9b1aARin3TLpc3yxNweWFzrozuCSm GnYNeNNy8OjdvPFw3/uue0qCY6vux7LlhCALbK4pGq58BFz2p7JZz7Um dCN3AnraZXWMhkG80d0ovafSyqOLPwBMg6rGXJyQnvFDkA2Y46ClZhOz r6PU3UvTEmtsa1IDaG8UeDdySojtSmMjSqaEepy7US86Gg==
M9U1TMUFRAKIF7VO0K21BGA5UV6K7GI8.com. 86400 IN NSEC3 1 1 0 - M9U2D1URHH6D8N46MEH6I21IHJK0LF7N NS DS RRSIG
M9U1TMUFRAKIF7VO0K21BGA5UV6K7GI8.com. 86400 IN RRSIG NSEC3 8 2 86400 20230311065138 20230304054138 36739 com. WERfD2ejQtwckrfEgvI+8HPTmXXI75SgJiiw4ypLyh+zBg2gVud2oOxi Zb7KjgCMiR0IfN4kTe7dFu5Sfv3v9TWQVevcwpZv2tc+aXAyCt6FBIPB qJAZ5bKEubSzCa5k0bkjZk6GZpLbJziOKv45QHqTiC/bD86c/tlOS30Q CV7mQXR47G3yCMBmhfXBEOYa5mxwrpqEgp5sQOiXz49Jdg==
;; Received 706 bytes from 2001:503:d2d::30#53(k.gtld-servers.net) in 35 ms

www.reditt.com.         3600    IN      A       103.224.182.222
;; Received 59 bytes from 103.224.212.6#53(421.ns2.above.com) in 64 ms

[23.01-RELEASE][admin@sg4860.local.lan]/: 

if we add that time up its a total of 88 ms.. There is no possible way that 0.088 seconds is slowing down your browsing of the internet ;) Or some backup software trying to resolve some fqdn.

If unbound is constantly restarting this can cause problems - for one it clears the full cache, and 2nd when you go to ask for something maybe unbound isn't even running..

Maybe your IPv6 is slowing down something, or causing delays, etc.

But the MYTH that resolving in any way amounts to slow internet is just that a MYTH.. Even if was talking 300ms to resolve anything - this is not an amount of time that would cause a local dns client to retrans a query or timeout, and you sure and the hell not going to notice a web page loading 300ms slower ;)

And again this would only the first time you looked up some all the way down from roots, after that is only going to be to the authoritative NS for whatever domain, or most likely just from cache.

If you are having problems with resolving for whatever reason - to correct it you need to start looking at specifics.. Are you having trouble talking to a NS in the path when looking up xyz? What is the exact resolve time use dig, or nslookup in debug mode to get this sort of info. Is unbound restarting, maybe enable full logging of queries and replies in unbound, etc. If IPv6 is possible problem, disable it in unbound for testing, etc. simple "do-ip6: no" in unbound custom option can disable that.

All I can say is I am on 23.01 and have had zero dns problems - ZERO.. Until this morning when I needed to restart unbound to test a ULA question someone else had, unbound had been running for 13 days without a restart. I use pfblocker - no issues with it either. Now I will give you that I use it in a limited fashion, only for maintain of my native aliases that contain stuff I want to block in my firewall, or allow via geoip lists, etc.

But to get to the root of whatever problem you experiencing we need to dig into specific details of what exactly is happening.

[Draco]

@johnpoz correct me please if I am wrong, but the dig you show, I believe, is just for the initial resolution of Reddit DNS name/IP address. I’m getting to the initial website with a fairly short delay, but then things will, on Flickr for instance, hang for quite some time. I don’t have the tools to see what’s going on in that HTTPS connection, but if you look at the page source, there is a lot of “asset loading“ from other URLs. I don’t think dig picks that up, and I think that is part of what is killing my performance (until the cache fills).

A similar problem happens on my back up program. If I rerun it in the foreground after it, fails, it completes without issue.

I’m not sure why you think this is related to my Internet connection or DNS configuration, when none of that changed with my switch to 23.01?

Please don’t get me wrong. I truly appreciate your help! I just don’t see a way to capture timings for all of the DNS look-ups as each program (or website) runs for the first time, or to emulate that. perhaps clearing my DNS logs and turning on more detailed unbound logging would capture that? I’m not sure.

Before I posted, I did look at the logs for Unbound (and all the pfBlockerNG logs, which looked fine), and Unbound was not restarting all the time. Only when pfBlockerNG ran its Cron job and there were changes to the IP block lists or DNS blacklists/whitelists.

The simplest way I can think of to test this is to go back to the 22.05 configuration. If the problems are gone, then it’s pretty certain the problem is tied to something that changed in 23.01. If that doesn’t fix it, I can quickly repave to 22.05 and continue testing.

[johnpoz]

@draco said in 23.01 breaks DNS resolver and pFblocker:

Flickr for instance, hang for quite some time

Ok how long does that take, in your browser web tools or whatever look to see what is trying to be resolved..

Again if your trying to troubleshoot something need specifics.. If fickr is a problem - what on that is the hold up..

That was even faster than reddit

; <<>> DiG 9.18.8 <<>> www.flickr.com +trace
;; global options: +cmd
.                       76294   IN      NS      l.root-servers.net.
.                       76294   IN      NS      a.root-servers.net.
.                       76294   IN      NS      c.root-servers.net.
.                       76294   IN      NS      j.root-servers.net.
.                       76294   IN      NS      k.root-servers.net.
.                       76294   IN      NS      h.root-servers.net.
.                       76294   IN      NS      g.root-servers.net.
.                       76294   IN      NS      b.root-servers.net.
.                       76294   IN      NS      d.root-servers.net.
.                       76294   IN      NS      f.root-servers.net.
.                       76294   IN      NS      m.root-servers.net.
.                       76294   IN      NS      e.root-servers.net.
.                       76294   IN      NS      i.root-servers.net.
.                       76294   IN      RRSIG   NS 8 0 518400 20230317050000 20230304040000 951 . kJaY9uAyEtbTnrjZ1qAQTsqHExUgSViSqmXstFQXUmBOgAbAHKQlp9Nj BCAb0pUbm3sDWOrGvOaqxN6QKFXd8331v6lxtsDKd3kIGE5Wo7kLwzw4 XzZeGRwfuRPnmwXtfYnJTo+X4tGgg2xK6c0uy5QdsFVzHEPwJNXURZVE rXQ/erzXJmKUXFuZim8sfm7UjTTJJsBwk8+P8uM+B9CKDtfE0CvxtyIS BbGi9pg4PDlJz0zB3V9VM/9+IcJuQ4NfnBDvw3pD9Q0LVx9qN2GzG1TK 06r6LMEBB9RRhO5wkZ7UwZuVzloYxntIpBMVL3zdTl2vVCIQFlzqSqJL 5OfW1A==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20230317170000 20230304160000 951 . hxP6AHA8/MhX3JTy2BcSkd4CeviA+3lw1LFWfIPNDgpdka84SUuKYc50 8hhh7bcW5/MDeKZJ82JkxBlZkrWWaNpGncQKOLdjmlkesYB03WPoOo/I aJohqNzLawFsVK4+2c48yrCeX1uesJQCiJnvJEUHyJmd8KtrRYeUnqDn nBiIlzuEHm5r3TQodZTO8AiH+Dp722SzlP8E8JI8LPdsozvClNKTGcCp KZVPMq3yCeuZA8+T859Ah8HuJjyh4NAEIAQe2K4uuD9B2ZSCt9lEf5i1 qcBwMXtUf9Od86hnXK/cjI6uCNMCPBBeN6QJ7uIQK64zHBZhejcPq0EN PU5D9w==
;; Received 1205 bytes from 2001:500:12::d0d#53(g.root-servers.net) in 45 ms

flickr.com.             172800  IN      NS      ns-573.awsdns-07.net.
flickr.com.             172800  IN      NS      ns-421.awsdns-52.com.
flickr.com.             172800  IN      NS      ns-1683.awsdns-18.co.uk.
flickr.com.             172800  IN      NS      ns-1244.awsdns-27.org.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20230311052252 20230304041252 36739 com. exNISCQI4v/S0m9ksCZH3zghILb9b1aARin3TLpc3yxNweWFzrozuCSm GnYNeNNy8OjdvPFw3/uue0qCY6vux7LlhCALbK4pGq58BFz2p7JZz7Um dCN3AnraZXWMhkG80d0ovafSyqOLPwBMg6rGXJyQnvFDkA2Y46ClZhOz r6PU3UvTEmtsa1IDaG8UeDdySojtSmMjSqaEepy7US86Gg==
8AEGLV925R77BHJM7FFD4RKA8CGTNSFK.com. 86400 IN NSEC3 1 1 0 - 8AEGTREIKABQ6N53PE432PFN3BMU2HM1 NS DS RRSIG
8AEGLV925R77BHJM7FFD4RKA8CGTNSFK.com. 86400 IN RRSIG NSEC3 8 2 86400 20230309061449 20230302050449 36739 com. kejD8AEDs1s8jUO2xUTJ2IN6Bgh2A5ItECrYExvbbQYZzSSnlbPEzyL7 n6uDtlE6TrYpOU/uH4wM+0Pt/USS4EmSUty+07+RF4hoM512BfYkUjxj QpQoLeFTRh3oFtFUQfQgYPD5oVJOtFcGErUhJ3lz3J4y9yavaa9phYxu Web4Fx3MJlvsA67u7Kp9NlrfTiF0JXHfqXBLyhXDbWM+zQ==
;; Received 745 bytes from 192.55.83.30#53(m.gtld-servers.net) in 19 ms

www.flickr.com.         60      IN      A       99.84.171.73
flickr.com.             300     IN      NS      ns-1244.awsdns-27.org.
flickr.com.             300     IN      NS      ns-1683.awsdns-18.co.uk.
flickr.com.             300     IN      NS      ns-421.awsdns-52.com.
flickr.com.             300     IN      NS      ns-573.awsdns-07.net.
;; Received 196 bytes from 205.251.196.220#53(ns-1244.awsdns-27.org) in 27 ms

[23.01-RELEASE][admin@sg4860.local.lan]/: 

And yes with dig that is a FULL resolve - this would be the slowest lookup of anything, because its a full resolve, down from the roots.. Even once the ttl for www.flickr.com expires - which 60 second ttl is just Fing insane.. You would only have to go talk to the NS for flickr.com directly..

Going back to 22.05 doesn't really tell you what the problem is - keep in mind unbound changed from like 1.15 to 1.17.1

I would troubleshoot the exactly problem vs rolling back all of pfsense.. Which really gives you no where to even start to what the actual problem is..

I can tell you right now there is nothing wrong with 23.01 or unbound 1.17.1 at least how I have mine configured.. Because again I have zero issues. maybe something specific with your hardware, your config, your connection, etc..

You know when you rollback - is when you are in a limited change window to update.. And something is not working and the change window is expiring.. And you hit the rollback mark, that is when you rollback ;)

There is not really a website on the planet anymore that loads just www.domain.tld -- they are all going to load sub domains or other resources off other domains, etc.. So call up your browser tools... How long does the page take to load?

I am not seeing any issues with www.flickr.com loading.. But then all I get is a page saying start for free.. What exactly are you loading..

If call up the browser console - I see quite a few "errors" etc.. but nothing look dns related - and the page popped pretty much instant and the background keeps changing pictures instantly.

If I view what is going on in the network and how long stuff takes - I see my ad blocker is blocking some stuff

But don't see anything failing from dns, or time out, etc.

Click the timing button - what does it show for dns resolution.. Clear your browser cache, restart unbound so its cache is clear, clear your os cache as well windows ipconfig /flushdns etc..

[terryzb]

@johnpoz As a pfSense and networking newbie, I just wanted to thank you John for your in-depth explanations, with pictures even! Much appreciated!

[areckethennu]

@johnpoz said in 23.01 breaks DNS resolver and pFblocker:

@gertjan good settings, another one I might suggest is serve zero. I have been using it for years, and have never had a problem with it.

serve 0 allows unbound to serve up the last IP it had in cache for some fqdn even if the ttl has expired. The ttl it hands off to the client will be 0, so if for some reason that doesn't work the client will have to ask again for that fqdn, and by this time unbound will have looked it up.

An aside from the crux of this thread, but I saw that and I'd like to try it. Is this the actual entry? It looks like it, but I thought I'd ask:

Serve Expired [] Serve cache records even with TTL of 0
When enabled, allows unbound to serve one query even with a TTL of 0, if TTL is 0 then new record will be requested in the background when the cache is served to ensure cache is updated without latency on service of the DNS request.

[johnpoz]

@areckethennu yeah that is the serve 0 setting.. Check the box..

[llebgrate]

Just upgraded to 23.01 myself and am experiencing DNS resolver issues. A quick restart of the service fixes it for clients. Wish I had saved the nslookup error on a domain I was testing. Can't recall the exact wording, but it was one I had not seen before about a server error.

I did enable "Serve Expired" in the advanced DNS resolver settings and will monitor to see if it happens again.

[Draco]

@johnpoz What I see when I log onto Flickr:

This site can’t be reached
www.flickr.com’s server IP address could not be found.
Try:
• Checking the connection
• Checking the proxy, firewall, and DNS configuration
ERR_NAME_NOT_RESOLVED

Then after 30+ seconds I see the Flickr homepage frame sans content, and things still appear to be loading after 2 minutes (but do not complete). Once I hit refresh (using Chrome), it loads right up, no problems.

When I open the Chrome debug window (note that I do not see the same thing you show in your screenshots, e.g. no TIMINGS menu) the timings on the initial page load don't seem too bad:

I am not seeing the 0 time for DNS; I suppose because it was not yet cached (I restarted Unbound and cleared my local DNS cache). But then, things get ugly when some fonts and scripts try to load:

The flickr.com load time is what is detailed above. The font failures are not, I suspect, fatal. I think the problem is the combo?yui:3.16.0/yui.../loader-hermes/... line. This references combo.static.flickr.com. When this load fails (seems to be jscript?), it torpedoes the rest of the website load (and accounts for the hang I see before refresh). My guess is this script does some init work and then loads other scripts.

When I hit REFRESH in the "hung" browser window, Flickr loads as noted above. The big difference is combo?yui?3.16.10…. call to that first Hermes URL (whatever that is -- scripts is my guess, as noted above) succeeds, and whatever that loads leads to a long list of more Hermes calls that also succeed. The website now loads properly.

I tried doing DIGs on flickr.com and combo.static.flickr.com (the URL for the Hermes URLs). They do not look alike, and I do not know enough about DIG to interpret that (happy to upload if you want a look).

I tried running these same tests on my older computer (slower), and I never run into the website load problems. My best guess is that my newer, faster computer is timing out. I did Malware and AV scans on my newer computer to ensure that was not causing issues. So the computer used to load the website makes a difference. I have replicated these problems using both Chrome and Firefox, and on multiple websites (see ironic note below). I've only drilled deeper on Flickr.

FWIW, when I run a Windows console app to do simple DNS queries (not DIG, just one call), I've seen these fail for google.com and other common sites too. This is why I suspected DNS issues. It now appears that it is not quite that simple, and is tied to machine speed somehow (or software config, or both or...?).

At this point I am well outside my toolset and knowledge (give me local code and a debugger and it's a different story), and have spent far more time than I have to spare on this issue. I can't have a production machine failing to load websites, and worse failing to do online backups or software updates, for the sake of finding this problem. If Netgate wants to get involved, then I'd be happy to set aside some time to work through this with them (I have the current 23.01 config on another USB stick). I remain hopeful that going back to 22.05 will have my production environment working again.

I do want to thank you for the time and effort you put into your responses. Without your screenshots and comments I would not have tried DIG or getting the timings included here.

On the side of irony, when I loaded the forum to enter this response, I hit the same This site can't be reached error. I think the difference between this and Flickr is that the forum isn't loading a script whose failure to load leaves the website non-functional a la Flickr. After a few seconds, the Netgate forum site just loads up and resolves normally. Given my tests with the DNS Query code I wrote (see google.com DNS query failure noted above), it is clearly related to DNS and timing issues; I just cannot pinpoint how.

[Draco]

@draco Replying to my last post: I decided to try a reboot from the Console before re-applying 22.05. Someone mentioned rebooting on another thread, which I had not tried because pfSense reboots as part of the upgrade. But I tried it anyhow...

So far my SG-5100 has been up for almost an hour and I have not repro'd the Flickr problem. What would rebooting change that leaves things working all of a sudden? My primary PC was not rebooted or changed, just the SG-5100.

[llebgrate]

Just following up that I tried the Serve Expired setting and a simple reboot and unfortunately the problem still persists.

windows client nslookup:

forum.netgate.com
Server: firewall.blah.com
Address: 192.168.150.1
*** firewall.blah.com can't find forum.netgate.com: Server failed

restart DNS Resolver service

forum.netgate.com
Server: firewall.blah.com
Address: 192.168.150.1

Non-authoritative answer:
Name: forum.netgate.com
Addresses: 2610:160:11:18::199
208.123.73.199

I did have telegraf scraping stats from the resolver as well, but have since turned it off and will continue to monitor.

[stephenw10]

No errors in the resolver log when it's failing to resolve?

If you turn up the logging does it at least show the incoming requests?

[llebgrate]

@stephenw10

I actually just caught it again. An entire page filled with these:

Mar 6 20:18:33	unbound	84264	[84264:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Mar 6 20:18:32	unbound	84264	[84264:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Mar 6 20:18:32	unbound	84264	[84264:3] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Mar 6 20:18:32	unbound	84264	[84264:3] info: generate keytag query _ta-4f66. NULL IN

Found this: https://forum.netgate.com/topic/152338/unbound-failed-to-prime-trust-anchor-could-not-fetch-dnskey-rrset-dnskey-in and am thinking it's dnssec related. I do in fact have forwarding and dnssec enabled, so going to play with the settings for a bit. Might also mess with dynamic dhcp client reg options. Haven't changed anything here though since the upgrade in a very very long time.

[stephenw10]

Hmm. Yes, that's DNSSec. I would at least try disabling and see if that removes the issue.
That really shouldn't be a problem though...