Slow DNS after 22.05

Gertjan

The fact that there's been a significant and open unbound bug since April
.....
...upon pfSense releases that there's an inclusion of known open issues

This one : SERVFAIL problems with unbound 1.15.0 running on OpenBSD 7.1 == https://github.com/NLnetLabs/unbound/issues/670

I can only say : it looks like what's descibed there.
For me : Itr's an OpenBSD thing. If it was an 'any' BSD bug, then why specify OpenBSD ?
The bug was also closed back in Avril 2022.
Also : I'm using the IPv6, and do not have any issues what so ever.

What should Netfgate have to do : list every closed bug from an external package in the past as "maybe not solved yet" ? That would be thousands of entries.

I saw you posted you posted to the bug report @unbound.
You should do what has been asked many times over there : you should add complete (very detailed) detailed unbound logs, so the author can see what's up and confirm what happened.
Right know, they (the author) will say : use the unbound version with the merged solution
included, and that's not possible right now.

All this IMHO of course.

pajinha

@mihaifpopa said in Slow DNS after 22.05:

Anyone else experiencing this?

This has been an amazing post... I got my issues fixed with the contributions of everyone, and in that process I got to learn how to debug dns unbound issues and get IPv6 working in my lab.

@Gertjan contributions have been great - made me want to start looking at Server Monitoring with Munin.

tentpiglet

@mikymike82

Can you verify that your IPv6 setup is correct?

ip6 is functioning. My wan has a 2001: address and clients on my network have a 2601: address. I can ping 2001:4860:4860::8888 from any of my network clients.

Mikymike82

@tentpiglet did you try the tests on the suggested websites as well…. This will sometimes give you a bit more insight.

lohphat

@gertjan The bug may have been closed in April but the issue still remains open in the 22.05 pfSense distribution since it's the unpatched unbound.

The issue should remain in an "Known Issues" list until the fix makes it into the next pfSense release. If that's not going to happen, then the process which NetGate uses to determine release viability needs to review upstream issues before release if it's not going to compile them for customers to review.

If the business model relies on components from upstream providers then doing some legwork to determine you're not inheriting problems sight-unseen seems reasonable. e.g. if there are included 3rd party modules not maintained by the OpenBSD distro (e.g. dhcpd, dpinger, igmpproxy, ntpd, radvd, sshd, syslogd, unbound, watchdogd) which are installed by default, then the "What's changed" notes of each should be reviewed by the release team and see what was changed, and if the versions included now have subsequent issues discovered before you pass the new versions on to us.

In all the software companies I've worked in, the release team took care of watching dependencies for any OSS (or commercial) component we then redistributed in our products.

Gertjan

@lohphat
I fully agree with you.
What happened if I was testing these new versions ?
For me, on my Netgate SG 4100, unbound works fine, and I'm using IPv6. half of all DNS requests go out over IPv6.
For me, Unboud runs solid for day, and get restarted because my pfBlockerng-devel reloads it after a week ( I'm not updating non updated feeds every hour or so).

I would have said to the Netgate team : for me, these new versions, like unbound, are ok.

The thing is : there are people using settings, or hardware, that differs from what Netgate used to test.
There is situation where the error pops up.
As we all us the exact same same unbound binary code, and the same pfSense code, only our settings can differ. And our uplink ....

mvikman

I don't have any DNS issues, but just out of curiosity regarding unbound settings...

I assume that the config file is /var/unbound/unbound.conf, because the the custom options get added to that file if set via resolver settings.

Here is my IPv6 system options, also no custom options set under DNS Resolver

When looking at the beginning of the unbound.conf file, the "do-ip6" is set to "no"...

##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-glue: yes
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes
do-daemonize: yes

Does pfSense actually also disable unbound IPv6 when IPv6 is disabled from System/Advanced settings?
Of course I could have tested it myself, but I didn't want to mess with my working system...

This with pfSense Plus 22.05

johnpoz

@mvikman do you have an actual IPv6 address?

here is from my config

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: no
hide-version: no
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes

See lower in the config is where your options get set and can override what is set there.

# Unbound custom options
server:
do-ip6: no
private-domain: "plex.direct"
local-zone: "use-application-dns.net"  always_nxdomain

mvikman

@johnpoz

I have IPv6 disabled and I don't have IPv6 address, my ISP doesn't support it.

The current unbound config file doesn't have that custom options section, because I haven't set any custom options.
But I tested adding the custom options and it does add them in the config file.

Just curious about that unbound's "do-ip6" is set to "no" without using custom options to set it.

johnpoz

@mvikman said in Slow DNS after 22.05:

"do-ip6" is set to "no" without using custom options to set it.

pretty pointless to do IPv6 if you don't have IPv6..

i unchecked that box and yes if you save a config on unbound then it sets that to no..

[22.05-RELEASE][admin@sg4860.local.lan]/root: cat /var/unbound/unbound.conf
##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: no
hide-version: no
harden-glue: yes
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes

mvikman

@johnpoz said in Slow DNS after 22.05:

@mvikman said in Slow DNS after 22.05:

"do-ip6" is set to "no" without using custom options to set it.

pretty pointless to do IPv6 if you don't have IPv6..

i unchecked that box and yes if you save a config on unbound then it sets that to no..

Yeah, that makes sense.

While reading this thread, I just somehow got stuck with the thought that to "fully disable" IPv6, that in addition to unchecking "Allow IPv6" in advanced settings, you would need to set the "do-ip6: no" to custom options. XD

johnpoz

@mvikman You don't have to disable IPv6 - you just need to keep unbound from using it as a transport.

vbredjp

having problems on 22.05 when DNS sometimes just stop resolving, but only certain domains (sometimes obscure domains) so it is hard to notice, as other domains are resolved ok.
do-ip6:no did not solve the problem.

serbus

This post is deleted!

vbredjp

This problem is getting unbearable, considering rolling back to previous version, what is the schedule for new pfsense+ release? as it may have newer unbound version with problem fixed..
I think I am being impacted by this bug
https://github.com/NLnetLabs/unbound/issues/670

but setting do-ip6: no does not solve problem for me.

SteveITS

@vbredjp said in Slow DNS after 22.05:

what is the schedule for new pfsense+ release

The next version will be 22.11, so presumably at least 3-4 months away.

I admit to not reading every post in detail, but for those seeing this, when it happens does restarting the DNS Resolver service clear it?

bmeeks

I believe there is more than a single unbound bug at work here. When you look at the commit history on the unbound GitHub repo and in the Change Log, you see a number of changes that are now rolled up into the latest 1.16.2 version of unbound.

It's a bit of an unfortunate timing thing that resulted in the current version of CE (2.6.0) having a much older unbound package (1.13.2) that is not impacted by the current bugginess of 1.15.0 (the version currently packaged with pfSense Plus 22.05).

So the issue appears to be first limited to just pfSense Plus installations, but certainly not all of them. CE users are running a much older unbound package and appear to not be suffering from this bug.

One thing the Netgate team could consider is bringing the current 1.16.2 version of unbound into their 22.05 FreeBSD Ports tree and thus making it available for manual installation (or upgrade) for those users impacted by the bug. Or put the older 1.13.2 version into the pfSense Plus 22.05 package repo. That would be a more complicated "update" for users, though, as they would likely need to manually remove the unbound package and then add it back as pkg would not normally see 1.13.2 as an "upgrade" for 1.15.0.

vbjp

@steveits
I have this problem really hard to troubleshoot as it impacts only certain domains not resolving at sporadic times. Restarting unbound service solves the problem for a while, but it's not sustainable as only yesterday I had to restart unbound 4 times.

vbjp

@bmeeks said in Slow DNS after 22.05:

One thing the Netgate team could consider is bringing the current 1.16.2 version of unbound into their 22.05 FreeBSD Ports tree and thus making it available for manual installation (or upgrade) for those users impacted by the bug. Or put the older 1.13.2 version into the pfSense Plus 22.05 package repo. That would be a more complicated "update" for users, though, as they would likely need to manually remove the unbound package and then add it back as pkg would not normally see 1.13.2 as an "upgrade" for 1.15.0.

This would be great people who don't have any problems could stay on default unbound version, and these with problems could manually install latest.
I unfortunately live in Japan and here lots of users like yahoo.co.jp from my observations this is worst offender, breaks 1 to 5 times a day, also other sites in co.jp break way more than com, net etc. Yesterday had amazon.co.jp break but amazon.com working perfectly. rebooting unbound everyday is getting a chore. Waiting 2-3 months until 22.11 release would be hell.

yellowRain

I have major issue with unbound on PfSense Plus latest stable version :

DNS lookups are slow because unbound (the DNS Resolver) frequently restarts

I do not know what to do, as system logs show no useful information

I'm with unbound 1.15.0.

I see strange hotplug events regarding igc0 in the General tab simultaneously, which may cause new dhcp lease and unbound restart.

Should I open a ticket, is it yet another intel nic driver / hardware issue ?

(SG-6100)