Slow DNS after 22.05

Mikymike82

@tentpiglet did you try the tests on the suggested websites as well…. This will sometimes give you a bit more insight.

lohphat

@gertjan The bug may have been closed in April but the issue still remains open in the 22.05 pfSense distribution since it's the unpatched unbound.

The issue should remain in an "Known Issues" list until the fix makes it into the next pfSense release. If that's not going to happen, then the process which NetGate uses to determine release viability needs to review upstream issues before release if it's not going to compile them for customers to review.

If the business model relies on components from upstream providers then doing some legwork to determine you're not inheriting problems sight-unseen seems reasonable. e.g. if there are included 3rd party modules not maintained by the OpenBSD distro (e.g. dhcpd, dpinger, igmpproxy, ntpd, radvd, sshd, syslogd, unbound, watchdogd) which are installed by default, then the "What's changed" notes of each should be reviewed by the release team and see what was changed, and if the versions included now have subsequent issues discovered before you pass the new versions on to us.

In all the software companies I've worked in, the release team took care of watching dependencies for any OSS (or commercial) component we then redistributed in our products.

Gertjan

@lohphat
I fully agree with you.
What happened if I was testing these new versions ?
For me, on my Netgate SG 4100, unbound works fine, and I'm using IPv6. half of all DNS requests go out over IPv6.
For me, Unboud runs solid for day, and get restarted because my pfBlockerng-devel reloads it after a week ( I'm not updating non updated feeds every hour or so).

I would have said to the Netgate team : for me, these new versions, like unbound, are ok.

The thing is : there are people using settings, or hardware, that differs from what Netgate used to test.
There is situation where the error pops up.
As we all us the exact same same unbound binary code, and the same pfSense code, only our settings can differ. And our uplink ....

mvikman

I don't have any DNS issues, but just out of curiosity regarding unbound settings...

I assume that the config file is /var/unbound/unbound.conf, because the the custom options get added to that file if set via resolver settings.

Here is my IPv6 system options, also no custom options set under DNS Resolver

When looking at the beginning of the unbound.conf file, the "do-ip6" is set to "no"...

##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-glue: yes
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes
do-daemonize: yes

Does pfSense actually also disable unbound IPv6 when IPv6 is disabled from System/Advanced settings?
Of course I could have tested it myself, but I didn't want to mess with my working system...

This with pfSense Plus 22.05

johnpoz

@mvikman do you have an actual IPv6 address?

here is from my config

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: no
hide-version: no
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes

See lower in the config is where your options get set and can override what is set there.

# Unbound custom options
server:
do-ip6: no
private-domain: "plex.direct"
local-zone: "use-application-dns.net"  always_nxdomain

mvikman

@johnpoz

I have IPv6 disabled and I don't have IPv6 address, my ISP doesn't support it.

The current unbound config file doesn't have that custom options section, because I haven't set any custom options.
But I tested adding the custom options and it does add them in the config file.

Just curious about that unbound's "do-ip6" is set to "no" without using custom options to set it.

johnpoz

@mvikman said in Slow DNS after 22.05:

"do-ip6" is set to "no" without using custom options to set it.

pretty pointless to do IPv6 if you don't have IPv6..

i unchecked that box and yes if you save a config on unbound then it sets that to no..

[22.05-RELEASE][admin@sg4860.local.lan]/root: cat /var/unbound/unbound.conf
##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: no
hide-version: no
harden-glue: yes
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes

mvikman

@johnpoz said in Slow DNS after 22.05:

@mvikman said in Slow DNS after 22.05:

"do-ip6" is set to "no" without using custom options to set it.

pretty pointless to do IPv6 if you don't have IPv6..

i unchecked that box and yes if you save a config on unbound then it sets that to no..

Yeah, that makes sense.

While reading this thread, I just somehow got stuck with the thought that to "fully disable" IPv6, that in addition to unchecking "Allow IPv6" in advanced settings, you would need to set the "do-ip6: no" to custom options. XD

johnpoz

@mvikman You don't have to disable IPv6 - you just need to keep unbound from using it as a transport.

vbredjp

having problems on 22.05 when DNS sometimes just stop resolving, but only certain domains (sometimes obscure domains) so it is hard to notice, as other domains are resolved ok.
do-ip6:no did not solve the problem.

serbus

This post is deleted!

vbredjp

This problem is getting unbearable, considering rolling back to previous version, what is the schedule for new pfsense+ release? as it may have newer unbound version with problem fixed..
I think I am being impacted by this bug
https://github.com/NLnetLabs/unbound/issues/670

but setting do-ip6: no does not solve problem for me.

SteveITS

@vbredjp said in Slow DNS after 22.05:

what is the schedule for new pfsense+ release

The next version will be 22.11, so presumably at least 3-4 months away.

I admit to not reading every post in detail, but for those seeing this, when it happens does restarting the DNS Resolver service clear it?

bmeeks

I believe there is more than a single unbound bug at work here. When you look at the commit history on the unbound GitHub repo and in the Change Log, you see a number of changes that are now rolled up into the latest 1.16.2 version of unbound.

It's a bit of an unfortunate timing thing that resulted in the current version of CE (2.6.0) having a much older unbound package (1.13.2) that is not impacted by the current bugginess of 1.15.0 (the version currently packaged with pfSense Plus 22.05).

So the issue appears to be first limited to just pfSense Plus installations, but certainly not all of them. CE users are running a much older unbound package and appear to not be suffering from this bug.

One thing the Netgate team could consider is bringing the current 1.16.2 version of unbound into their 22.05 FreeBSD Ports tree and thus making it available for manual installation (or upgrade) for those users impacted by the bug. Or put the older 1.13.2 version into the pfSense Plus 22.05 package repo. That would be a more complicated "update" for users, though, as they would likely need to manually remove the unbound package and then add it back as pkg would not normally see 1.13.2 as an "upgrade" for 1.15.0.

vbjp

@steveits
I have this problem really hard to troubleshoot as it impacts only certain domains not resolving at sporadic times. Restarting unbound service solves the problem for a while, but it's not sustainable as only yesterday I had to restart unbound 4 times.

vbjp

@bmeeks said in Slow DNS after 22.05:

One thing the Netgate team could consider is bringing the current 1.16.2 version of unbound into their 22.05 FreeBSD Ports tree and thus making it available for manual installation (or upgrade) for those users impacted by the bug. Or put the older 1.13.2 version into the pfSense Plus 22.05 package repo. That would be a more complicated "update" for users, though, as they would likely need to manually remove the unbound package and then add it back as pkg would not normally see 1.13.2 as an "upgrade" for 1.15.0.

This would be great people who don't have any problems could stay on default unbound version, and these with problems could manually install latest.
I unfortunately live in Japan and here lots of users like yahoo.co.jp from my observations this is worst offender, breaks 1 to 5 times a day, also other sites in co.jp break way more than com, net etc. Yesterday had amazon.co.jp break but amazon.com working perfectly. rebooting unbound everyday is getting a chore. Waiting 2-3 months until 22.11 release would be hell.

yellowRain

I have major issue with unbound on PfSense Plus latest stable version :

DNS lookups are slow because unbound (the DNS Resolver) frequently restarts

I do not know what to do, as system logs show no useful information

I'm with unbound 1.15.0.

I see strange hotplug events regarding igc0 in the General tab simultaneously, which may cause new dhcp lease and unbound restart.

Should I open a ticket, is it yet another intel nic driver / hardware issue ?

(SG-6100)

yellowRain

@yellowrain EDIT : I do not use the igc0 port anymore.
Just realized ix0 is smoother. Too bad I did not do that earlier.

Gertjan

@yellowrain said in Slow DNS after 22.05:

I see strange hotplug events regarding igc0 in the General tab simultaneously, which may cause new dhcp lease and unbound restart.

igc0 is your WAN ?
That would be a very valid reason for unbound, actually any process, that uses interfaces.
3 things to test : the cable. The interface on the other side, the igc0 from your 6100.
The cable test is easy ;)
You could use another WAN interface on your 6100, it has plenty of interfaces ;)
Testing the other side : use another NIC, if ythe upstream device has more then one, or put a switch between your WAN (igc0) and your upstream device. This will hide the problem, you still have to check why the upstream device pulls its interface down. If this is a modem type device, it does so because your uplink went down.

yellowRain

@gertjan my current config was igc0 for LAN. ix2/ix3 for WAN.
I had time to fully investigate all logs this summer.

the interface on the other side has been rock stable (trusty business grade switches Zyxel XG1930-10).

2.5gbps on igc0 is still not as stable as I would wish, based on my experience. (connection lost even at max power, green ethernet not working, short cable setting unreliable. That makes unbound restart and the restart process takes time)

Cable may be one reason, you're right, I had Cat 6A, though even another fully compatible 10gbps cable make the igc0 exhibit same symptoms (maybe less, but I rushed thoses tests...).

So at the end, for today, I use only ix0 for LAN, ix2/ix3 for WAN. Those interfaces are server-level Intel chipsets based. Other interfaces I have in my homelab are almost all X550 (NAS, server), and that works well. Only one exception is one aquantia thunderbolt 3 interface for my laptop, which is great also for this type of device.

I also fully investigated the wireless part, thanks to openwrt on wrt3200acm (6ghz wifi routers are still not widely available here in Europe). There I found some IoT smart plugs screwing 2.4Ghz network, on which devices land sometimes. I had to add a separate 2.4Ghz radio to isolate thes IoT devices. 5ghz was already optimized, but the latest 22.03 OpenWRT build brought long uptime stability (or at least uncluttered the logs).

That way, confcalls over Teams, voWifi calls and DNS resolution are now stable.