23.01 Upgrade unbound Issue

stephenw10

That's your own hosted DNS resolvers? Interesting. They support DNSSec I preseume.

It seems unlikely any upstream change, at Quad9, would have coincided with 23.01 release. But perhaps people just started checking after upgrading. I agree some behaviour change in Unbound seems most likely to me.

Steve

johnpoz

@stephenw10 I just did a breeze over of the changes and bug fixes in unbound from 1.15 to 1.17.1 and nothing jumped out at me.. But possible something that was fixed now triggers more failures that should of failed before but didn't, etc.

I despise forwarding, so while I will turn it on to show how it can be done. I have no desire to leave it on for any length of time to see if errors occur on specific domains I might be going to, etc. Or after a amount of time..

if someone could point out a specific query that fails when forwarding using dnssec and forwarding, but works fine with just forwarding to a dnssec resolver, be it over tls or just clear, etc. I would be up for looking into that.

bingo600

I'm only forwarding because:

1:
I already had a working linux DNS/DHCP setup , that does dynamic DHCP entries in DNS.

2:
Unbound DHCP adding "hosts", created unacceptable reload/delays.
I want my DHCP devices to be resolvable, and more important to have "sane names" in the DNS system.

DNSSEC all 20 meters on a local network ... Is kind'a "well it hasen't broke yet" , and my Bind9's are setup to accept it.

/Bingo

johnpoz

@bingo600 said in 23.01 Upgrade unbound Issue:

only forwarding because

Your forwarding to your own dns? What does it do then to resolve public dns. Forwarding to your own internal resolvers is different ball game than forwarding to some outside dns ;)

bingo600

@johnpoz said in 23.01 Upgrade unbound Issue:

@bingo600 said in 23.01 Upgrade unbound Issue:

only forwarding because

Your forwarding to your own dns? What does it do then to resolve public dns.

My bind9's knows all my DNS names & Dynamic (DHCP) DNS names , and will resolve public names if asked to do so.

That made the decision on setting pfSense to query those , super easy ....

On (almost) all of my lan's/vlan's - I only allow DNS (UDP/TCP 53) to the def-gw ip (pfSense), all other DNS servers are blocked.

So client's can only ask pfSense , that asks my linux'es , they resolve local names , and ask A-Root for the rest ...

My Phone Vlan + MMedia Vlan (ATV etc) , asks a Pi-Hole directly , that asks my linux'es.
It cleans out a lot of "noise" on phones , and blocks a lot of ATV "callbacks" , and gives me the possibility to block stuff manually.

But you use Pi-Hole so .. You know that :-)

I (hate DOH) .... Have floating rules on most IF's that "try to block" DOH .... , using internet based lists, that pfS downloads once a day.

Forwarding to your own internal resolvers is way different than forwarding to some outside dns ;)

I know , and might be why i'm not hit , and haven't removed DNSSEC
/Bingo

SteveITS

@johnpoz said in 23.01 Upgrade unbound Issue:

specific query that fails when forwarding using dnssec and forwarding

Not sure if it was this thread or another (and was in the redmine) but I posted "nslookup linkedin.com" failed for me, repeatedly. I restarted unbound while testing, to no avail. Unchecking the DNSSEC option (which I believe restarts unbound unfortunately, for A/B testing) let it work immediately. I was unable to replicate it later, but I did not leave it with DNSSEC enabled for several hours as it was originally. So it may be time based, or Microsoft may have changed something, or something else triggers it.

re: why forward, Quad9 and others add malware protection. No it's not necessary but is a value to some.

johnpoz

@steveits said in 23.01 Upgrade unbound Issue:

Quad9 and others add malware protection

That for sure could be a valid reason - but most browsers do that on their own to be honest.

https://support.mozilla.org/en-US/kb/how-does-phishing-and-malware-protection-work

Not saying better or worse than what a bad site filtering might do - but that feature is not worth me sending all my dns traffic to them. Other might find it worth while sure..

johnpoz

@steveits said in 23.01 Upgrade unbound Issue:

linkedin.com

They are not even using dnssec.. But I see this error checking their dns

linkedin.com/TXT: No response was received until the UDP payload size was decreased, indicating that the server might be attempting to send a payload that exceeds the path maximum transmission unit (PMTU) size. (2600:2000:2210::43, 2600:2000:2220::43, 2600:2000:2230::43, 2600:2000:2240::43, UDP_-_EDNS0_4096_D_KN)

bingo600

I have experienced in my setup , that "Unbound" caches a "non resolved answer too".

If i have forgotten to add ie. a new host to my zone , and i try to resolve it on a client. I get a "not found" as i should
But when i then add the host to bind9 , and retry the "lookup" , then i still get a "not found" from unbound.
If i try to resolve the name on the bind9 linux machine, it resolves fine.
Cure is to restart unbound , then it begins to work.

I know the "zones" specifies the TTL , but does it have to "cache a bad answer too" ...

I might have to dig into my "Cricket book" .....
Prob. to find out that TTL also applies to "non resolvables too, for that zone".

Well just don't GOOF in the first place ...

/Bingo

johnpoz

@bingo600 said in 23.01 Upgrade unbound Issue:

cache a bad answer

hmmm - I just did a quick look, and it doesn't seem like the gui exposes the

cache-max-negative-ttl:

Setting... this could have something to do with it.. Especially if you have min ttl set. There was a bug a while back I believe where even if you had cache-max-negative-ttl: set to say 1, sometimes this could get cached as your min ttl time vs the negitive ttl..

I will have to do a bit of looking at what the actual unbound.conf has - and what is default if not set. But maybe I missed it, but doesn't seem like that option is exposed in the gui to mess with even.

edit: ok I don't see it in the .conf - so looks like it would default to 3600 from the unbound documentation

   cache-max-negative-ttl: <seconds>
          Time to live maximum for negative responses, these have a SOA in
          the authority section that is limited in time.  Default is 3600.
          This applies to nxdomain and nodata answers.

You should be able to set that in the custom options for unbound, if its something you feel you want to adjust - maybe put in a feature request to expose that in the gui.

SteveITS

@steveits Also, to clarify, I do not think LinkedIn was the only issue at the time. I was on my phone and flipping between things. I vaguely recall some issues on web pages, and the LinkedIn app wouldn't load most things, which got my attention, especially since I'd just upgraded to 23.01 that evening. Their web site wouldn't load, so I started investigating. No further issues in the last few days after turning off DNSSEC. My testing was querying pfSense, I flushed DNS cache on my PC, etc.

johnpoz

@bingo600 ok did a bit of looking at the neg cache time..

So you can tell that neg cache is being used, if you looking up something you know will nx.. So for example I did a query for random.cnn.com

If I look in the unbound cache, I can see that its counting down the ttl from 3600

[23.01-RELEASE][admin@sg4860.local.lan]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep lsjfdsld.cnn.com
msg lsjfdsld.cnn.com. IN A 32899 1 3554 3 0 1 0
[23.01-RELEASE][admin@sg4860.local.lan]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep lsjfdsld.cnn.com
msg lsjfdsld.cnn.com. IN A 32899 1 3546 3 0 1 0
[23.01-RELEASE][admin@sg4860.local.lan]/var/unbound:

See where it goes down from 3554, to 3546.. You could try setting the min-neg cache setting to something lower and see if using that via similar test that I did.

moelassus

@bingo600 Which list are you using for DoH blocking? I'm currently using a list from "oneoffdallas" that appears to be maintained. Wondering if there is a better one.

SteveITS

@moelassus There's a TheGreatWall_DoH_IP list in pfBlocker, though the list itself says it was last updated in 2020.

moelassus

@steveits Yeah, TheGreatWall list doesn't appear to be maintained. For example, it doesn't include any of Apple's DoH servers. OneOffDallas's list was last updated in Dec 2022.

Trying to block DoH is really a pointless exercise because bad actors aren't going to use a well-known DoH server anyway. DoH is a scourge. ️

johnpoz

@moelassus said in 23.01 Upgrade unbound Issue:

going to use a well-known DoH server anyway

While I agree with you - it would be quite possible for some bad software to use something on their own, and not a well known doh server. What it does do is stop stuff that is just trying to do you a "favor" and use doh without specifically asking you.. Those would normally point to a well known doh service..

I block it - because I don't want my stuff using it, but sure if it just using some not well known doh server, not much I could do about that other than trying all of its https traffic, which is pretty difficult.

bingo600

@johnpoz said in 23.01 Upgrade unbound Issue:

You could try setting the min-neg cache setting to something lower and see if using that via similar test that I did.

Thanx .. I will try that

And it works ....

server:
#log-queries: yes
#log-replies: yes
# Set max failed lookup cache time
cache-max-negative-ttl: 10
#

[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 6 3 0 1 0
msg cnn.com. IN DS 33152 1 6 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 4 3 0 1 0
msg cnn.com. IN DS 33152 1 4 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 3 3 0 1 0
msg cnn.com. IN DS 33152 1 3 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 2 3 0 1 0
msg cnn.com. IN DS 33152 1 2 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 1 3 0 1 0
msg cnn.com. IN DS 33152 1 1 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
msg garbage.cnn.com. IN A 33155 1 0 3 0 1 0
msg cnn.com. IN DS 33152 1 0 0 0 3 0
[23.01-RELEASE][admin@fwall]/var/unbound: unbound-control -c /var/unbound/unbound.conf dump_cache | grep .cnn.com
[23.01-RELEASE][admin@fwall]/var/unbound:

bingo600

@moelassus said in 23.01 Upgrade unbound Issue:

@bingo600 Which list are you using for DoH blocking? I'm currently using a list from "oneoffdallas" that appears to be maintained. Wondering if there is a better one.

I was basically following this guid
https://github.com/jpgpi250/piholemanual

URL List defined in pfS

And one of the lists got updates today

@moelassus - There's a pfSense specific guide in this doc
https://github.com/jpgpi250/piholemanual/tree/master/doc

I just skimmed the new guide ... He has made it very complicated.
I have attached the guide i followed, when i made it .... (a previous version)
Tip : USE FLOATING RULES ...
1-pfs-blockDOH-2021-simple.pdf.zip

/Bingo

inferno480

Definitely still seeing some Unbound issues since upgrading to 23.01... sometimes it takes a couple days to happen, and you can "fix it" by restarting unbound but it appears to randomly crash/restart on its own then log messages about DNSKEYs being insecure.

I also had the issue of IPv6 interfaces not being auto-added to the ACL; I had to manually override the ACL and put in all of my v6 subnets / internal IPs which I thought was the only issue (error would be Query Refused) but I just had an incident where unbound crashed and gave "Server Fails" in nslookup w/the ACL fix in place. I did nothing but wait a few moments and it was working again.

Every server restart you see in this log was not caused by me and this self-fixed after several moments of trying various DNS lookups (suddenly they all worked). Not really sure what to do about this except revert back to 22.05.

Under DNS Resolver > Advanced Settings I have "Prefetch DNS Key support" and "Harden DNSSEC Data" both UNchecked (been unchecked since I had the Query Failed/ACL issues two days after I upgraded)

Feb 22 17:32:33 unbound 73817 [73817:5] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:32:33 unbound 73817 [73817:5] info: generate keytag query _ta-4f66. NULL IN
Feb 22 17:31:25 unbound 73817 [73817:2] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:31:25 unbound 73817 [73817:2] info: generate keytag query _ta-4f66. NULL IN
Feb 22 17:30:24 unbound 73817 [73817:2] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:30:24 unbound 73817 [73817:2] info: generate keytag query _ta-4f66. NULL IN
Feb 22 17:29:21 unbound 73817 [73817:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:29:21 unbound 73817 [73817:8] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:29:21 unbound 73817 [73817:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Feb 22 17:29:21 unbound 73817 [73817:8] info: generate keytag query _ta-4f66. NULL IN
Feb 22 17:29:21 unbound 73817 [73817:1] info: generate keytag query _ta-4f66. NULL IN
Feb 22 17:29:21 unbound 73817 [73817:0] info: start of service (unbound 1.17.1).
Feb 22 17:29:21 unbound 73817 [73817:0] notice: init module 1: iterator
Feb 22 17:29:21 unbound 73817 [73817:0] notice: init module 0: validator
Feb 22 17:29:21 unbound 73817 [73817:0] notice: Restart of unbound 1.17.1.
(...)

inferno480

@inferno480 I had to truncate the log to fit my post in, but can attach it... I just don't think there's much of value to review in it unless I increase debugging somewhere. i.e. it shows the symptom more than the cause.