DNS Resolver needs a constant reboot to work

scottlindner

@gertjan said in DNS Resolver needs a constant reboot to work:

(don't ask why an atom is known as amd ;) )

Gonna guess it's related to how software was packaged, built and labelled long ago when AMD and Intel had different builds and now they don't.

We're using both the same unbound version, unbound 1.15.0. Probably also the same pfSense Plus version 22.05.

Yup!

I'm using also the OpenVPN server.

You use VLANs : I can imagine that the slightest VLAN setting change on the SG 2220 can make your LAN unreachable.
Although : just restarting unbound wouldn't repair that. So I rule out VLAN issues.

Yah. I do too. And I rule out physical network issues. I have run Cat6 to all rooms and have multiple PoE APs in throughout the house to ensure everything has solid network. I feel this is truly a behavoiral/configuration type of thing but why now? Why not with earlier versions of pfSense? I have been running this Netgate since mid 2017 and this has been an issue for about a year or so.

There was a big 'what the heck is going on with unbound 0.15.0' forum thread a while back ago. I'll go over it, check if something matches your description.

Ya. I know a bunch of folks have had issues with unbound for various reasons. Appreciate you looking. I'm trying hard to find anything to correlate the issue with. I'm even losing confidence it's related to OpenVPN because of how quickly it happened yesterday after an unbound restart and there is nothing in the logs that makes any sense.

Gertjan

@scottlindner

This is the thread : Slow DNS after 22.0
It talks about 'buffers' and IPv6.

Btw : a thread like that, I hate it.
I've been trying a lot to create the same unbound behaviour.
I'm using an Intel Atom, a Netgate SG 4100 device, I do use IPv6 on my LANs, not that I'm really needing it.
I'm using pfBlockerng-devel, with a "restart reload feeds ones a week" as I don't want unbound to get restarted every 60 minutes or so, because some one somewhere added one DNSBL on some list I use.

The thing is : if we use the same hardware (a Netgate device), the some software, then what is different ?
Our LAN : the cables, switches and devices.

I guess, my unbound knows I'm watching him.
See here what I use so I can see what happens when. That's my unbound. It restart a lot, because, while I'm writing here, I do try stuff before posting, my unbound gets restarted.
But when I'm not poking in my pfSense, you can see unbound runs for days or weeks without a restart.
Of course, if unbound stops handling DNS for my LANs, credit card machines start to fail .... and then all hell breaks loose. As money stops coming in.

An there is of course Unbound 1.15.0 released and this version was replaced by 0.15.1 , 0.15.2 etc.
These sub versions came out because of 'issues'. pfSense doesn't allow us (easily) to try all these versions, its "whatever they chose" up until the next pfSense release comes out.

What 0.15.0 changed, according to nlnet labs, the author, was DNSSEC related.
What about shutting down DNSSEC entirely for a while ?

scottlindner

This post is deleted!

scottlindner

@gertjan said in DNS Resolver needs a constant reboot to work:

@scottlindner

This is the thread : Slow DNS after 22.0
It talks about 'buffers' and IPv6.

Btw : a thread like that, I hate it.
I've been trying a lot to create the same unbound behaviour.
I'm using an Intel Atom, a Netgate SG 4100 device, I do use IPv6 on my LANs, not that I'm really needing it.
I'm using pfBlockerng-devel, with a "restart reload feeds ones a week" as I don't want unbound to get restarted every 60 minutes or so, because some one somewhere added one DNSBL on some list I use.

The thing is : if we use the same hardware (a Netgate device), the some software, then what is different ?
Our LAN : the cables, switches and devices.

I guess, my unbound knows I'm watching him.
See here what I use so I can see what happens when. That's my unbound. It restart a lot, because, while I'm writing here, I do try stuff before posting, my unbound gets restarted.
But when I'm not poking in my pfSense, you can see unbound runs for days or weeks without a restart.
Of course, if unbound stops handling DNS for my LANs, credit card machines start to fail .... and then all hell breaks loose. As money stops coming in.

An there is of course Unbound 1.15.0 released and this version was replaced by 0.15.1 , 0.15.2 etc.
These sub versions came out because of 'issues'. pfSense doesn't allow us (easily) to try all these versions, its "whatever they chose" up until the next pfSense release comes out.

What 0.15.0 changed, according to nlnet labs, the author, was DNSSEC related.
What about shutting down DNSSEC entirely for a while ?

I had already disabled DNSSEC from a while ago when I was trying to bang on this issue before.

Is there a way to reset DNS Resolver settings to default? I couldn't find anything. I just want to be sure I haven't done something unintentional as I have been trying to get my Internet to work without having to man the pfSense console constantly.

scottlindner

Crap. I tried upgrading to the latest experimental just to try to force my way out of this problem and now upgrading is jacked too. It failed with meaningless "unable to..." errors and now the Update system fails to check for updates. Guess I gotta rebuild this thing... again^3.

If I save off the XML is there an effective way to hack the XML for a clean install but to make sure Unbound is set to defaults? I know I can just manually rebuild everything and maybe that's what I'll do. Getting the VLANs back to working is the pivotal part for me because until that happens I gotta use a laptop connected to the Netgate.

Gertjan

@scottlindner said in DNS Resolver needs a constant reboot to work:

Crap. I tried upgrading to the latest experimental

You went from 22.05 to something new ?
I wouldn't dare doing so.

Read some posts from here : Home > pfSense Software > Development should make you think otherwise.

@scottlindner said in DNS Resolver needs a constant reboot to work:

Guess I gotta rebuild this thing... again

You have a good backup up config file when you left 22.05 ? That's the one you need.
If not, take one from here : /cf/conf/backup : look at the date / time stamp, and use the one you where using with 22.05.

Installing a fresh 22.05, assign a minimal LAN + WAN + import xml + reboot and you'll be back to square one.

edit : extra info :

https://forum.netgate.com/topic/174248/need-help-troubleshooting-dns-after-upgrade-to-22-05

scottlindner

@gertjan said in DNS Resolver needs a constant reboot to work:

@scottlindner said in DNS Resolver needs a constant reboot to work:
Crap. I tried upgrading to the latest experimental 
You went from 22.05 to something new ?
I wouldn't dare doing so.

Read some posts from here : Home > pfSense Software > Development should make you think otherwise.

@scottlindner said in DNS Resolver needs a constant reboot to work:

Guess I gotta rebuild this thing... again

You have a good backup up config file when you left 22.05 ? That's the one you need.
If not, take one from here : /cf/conf/backup : look at the date / time stamp, and use the one you where using with 22.05.

Installing a fresh 22.05, assign a minimal LAN + WAN + import xml + reboot and you'll be back to square one.

edit : extra info :

https://forum.netgate.com/topic/174248/need-help-troubleshooting-dns-after-upgrade-to-22-05

I just reinstalled and restored the latest config. The update check is now working. I'm kinda hopeful my DNS Resolver issues just magically go away. I did save off a copy of the config XML in the default fresh installation state so I can hack an XML file to restore certain features to default. This being the key that I was looking for in this particular thread:

	<unbound>
		<enable></enable>
		<dnssec></dnssec>
		<active_interface></active_interface>
		<outgoing_interface></outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
	</unbound>

This is the unbound config that I restored.

	<unbound>
		<enable></enable>
		<active_interface>all</active_interface>
		<outgoing_interface>all</outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
		<port></port>
		<sslcertref>589929552f3cb</sslcertref>
		<regdhcpstatic></regdhcpstatic>
		<system_domain_local_zone_type>transparent</system_domain_local_zone_type>
		<tlsport></tlsport>
	</unbound>

Would any of those differences cause issues? Seems pretty trivial and more like skeleton config rather than influential.

scottlindner

@gertjan

One reason I love running my Netgate. Never would have gotten this with a traditional residential router.

scottlindner

This is interesting. After reinstalling pfSense and restoring my configuration things seem much more stable. I know it's still early to know for sure. I even shut off the reboot and unbound restart cron jobs and things are good so far. My previous pfSense installation was a lot of upgrades so I'm wondering if some configuration got left behind that a clean install took care of that doesn't show up in the XML but does on the device OS.

In the future when there is an update available for my Netgate, I'm going to do a clean install and config restore rather than upgrade.

t__2

I have the exact same problem here with a 5100. Pfsense stops providing DNS services to everything on our internal network. unbound has not stopped. It shows it is still running. I can do a ping from the Diagnostic menu and it resolves the IP just fine and the ping works but only from inside the pfsense gui. I have not tried it when ssh'd into the 5100 yet.

Fortunately for us it only does this every few weeks. So when it does I log into the gui and restart unbound and everything works again. I am also running OpenVPN. I might be retiring OpenVPN soon for Wiregaurd. We will see if that changes anything.

scottlindner

@t__2 said in DNS Resolver needs a constant reboot to work:

I have the exact same problem here with a 5100. Pfsense stops providing DNS services to everything on our internal network. unbound has not stopped. It shows it is still running. I can do a ping from the Diagnostic menu and it resolves the IP just fine and the ping works but only from inside the pfsense gui. I have not tried it when ssh'd into the 5100 yet.

Mine seems to be stable after a fresh install of pfSense. I saved off the config, installed fresh, loaded config and I seem to be fine now. Although I'm not going to say that with 100% confidence because sometimes it takes months.

I did have another similar issue but restarting Unbound didn't fix it but a reboot did. That happened twice so I set a Cron to reboot pfSense once a week and now I seem great. Again, waiting for long term to confirm.

t__2

@scottlindner Switched to wireguard removed OpenVpn for pfsense but still had the problem. Upgrade to 23.01 on 2023-04-17 and have not had to reboot unbound since. Maybe they fixed it?

scottlindner

@t__2
For me it wasn't the upgrade so much as a clean install after the upgrade. Has been totally fine since the flash install.