DNS Resolver needs a constant reboot to work

scottlindner · Dec 2, 2022, 4:17 PM

@gertjan said in DNS Resolver needs a constant reboot to work:

@scottlindner said in DNS Resolver needs a constant reboot to work:

....... I just want my internet to work.

And I believe you.
I'm running a company behind mine, that is, it's a hotel, so I share (captive portal) my connection with my clients.

Ya. So you have a lot more responsibilitis than I do, but it's also your paid job. This is my home network and as a single dad of three kids that play hockey and I coach all three of their teams, I really don't have the time anymore to be a network engineer like I used to. That is why I bought the Netgate to begin with because I wanted turnkey. It used to be super reliable and now it isn't. I haven't changed anything. It just became unstable. I have restarted from default several times and this issue keeps coming back so I'm feeling it's newer versions of pfSense... somehow.

@scottlindner said in DNS Resolver needs a constant reboot to work:

This is becoming unusable and I don't have a clue what to do to fix it.

That's why I was answering with 'some' details.
I guess you testing them will take you less time as me writing up the post.

I had started a reply appreciating the great detail you put in. Was doing some digging here first.

You clearly stated in you second post that resolving worked for pfSense, but not your desktop device.
That is, IMHO, uncommon.

Yah. That is what kinda blew me away. I didn't expect that at all.

If you do 'nothing', then at least : use 100 % Netgate Resolver default settings.
But un check "DHCP Registration" on that Resolver settings page, you probably understood by now why I say this.

It hasn't been checked. I haven't ever checked it. I havent touched DNS Resolver settings... ever. I am using pfSense almost completely default. I do have a hunch though related to some of the non-default stuff I have done. I do have two VLANs for separation of traffic. Basically IoT devices that just need Internet and everything else. The IoT network doesn't have access to pfSense GUI. But the computer I'm using is wired on the everything else VLAN. I have been doing this for five years, so it isn't like this is a sudden change. BUT... I have this hunch that it might be related to OpenVPN. Twice now it seems I have stability issues after hitting my home network from remote using OpenVPN on pfSense. It's just a hunch for now, but I am starting to suspect it is related to the OpenVPN Server I have configured.

Btw : is a 2220 a amd or arm device ?

Neither. It's an Intel Atom.

Gertjan · Dec 2, 2022, 4:33 PM

@scottlindner said in DNS Resolver needs a constant reboot to work:

Neither. It's an Intel Atom.

Like mine (Netgate SG 4100) : an atom is known as an 'amd' :

login-to-view

(don't ask why an atom is known as amd ;) )

We're using both the same unbound version, unbound 1.15.0. Probably also the same pfSense Plus version 22.05.

I'm using also the OpenVPN server.

You use VLANs : I can imagine that the slightest VLAN setting change on the SG 2220 can make your LAN unreachable.
Although : just restarting unbound wouldn't repair that. So I rule out VLAN issues.

There was a big 'what the heck is going on with unbound 0.15.0' forum thread a while back ago. I'll go over it, check if something matches your description.

bingo600 · Dec 2, 2022, 6:03 PM

@gertjan said in DNS Resolver needs a constant reboot to work:

(don't ask why an atom is known as amd ;) )

That's just the architecture it was compiled for.
And AMD was first w 64bit

/Bingo

scottlindner · Dec 2, 2022, 6:03 PM

@gertjan said in DNS Resolver needs a constant reboot to work:

(don't ask why an atom is known as amd ;) )

Gonna guess it's related to how software was packaged, built and labelled long ago when AMD and Intel had different builds and now they don't.

We're using both the same unbound version, unbound 1.15.0. Probably also the same pfSense Plus version 22.05.

Yup!

I'm using also the OpenVPN server.

You use VLANs : I can imagine that the slightest VLAN setting change on the SG 2220 can make your LAN unreachable.
Although : just restarting unbound wouldn't repair that. So I rule out VLAN issues.

Yah. I do too. And I rule out physical network issues. I have run Cat6 to all rooms and have multiple PoE APs in throughout the house to ensure everything has solid network. I feel this is truly a behavoiral/configuration type of thing but why now? Why not with earlier versions of pfSense? I have been running this Netgate since mid 2017 and this has been an issue for about a year or so.

There was a big 'what the heck is going on with unbound 0.15.0' forum thread a while back ago. I'll go over it, check if something matches your description.

Ya. I know a bunch of folks have had issues with unbound for various reasons. Appreciate you looking. I'm trying hard to find anything to correlate the issue with. I'm even losing confidence it's related to OpenVPN because of how quickly it happened yesterday after an unbound restart and there is nothing in the logs that makes any sense.

Gertjan · Dec 3, 2022, 1:18 PM

@scottlindner

This is the thread : Slow DNS after 22.0
It talks about 'buffers' and IPv6.

Btw : a thread like that, I hate it.
I've been trying a lot to create the same unbound behaviour.
I'm using an Intel Atom, a Netgate SG 4100 device, I do use IPv6 on my LANs, not that I'm really needing it.
I'm using pfBlockerng-devel, with a "restart reload feeds ones a week" as I don't want unbound to get restarted every 60 minutes or so, because some one somewhere added one DNSBL on some list I use.

The thing is : if we use the same hardware (a Netgate device), the some software, then what is different ?
Our LAN : the cables, switches and devices.

I guess, my unbound knows I'm watching him.
See here what I use so I can see what happens when. That's my unbound. It restart a lot, because, while I'm writing here, I do try stuff before posting, my unbound gets restarted.
But when I'm not poking in my pfSense, you can see unbound runs for days or weeks without a restart.
Of course, if unbound stops handling DNS for my LANs, credit card machines start to fail .... and then all hell breaks loose. As money stops coming in.

An there is of course Unbound 1.15.0 released and this version was replaced by 0.15.1 , 0.15.2 etc.
These sub versions came out because of 'issues'. pfSense doesn't allow us (easily) to try all these versions, its "whatever they chose" up until the next pfSense release comes out.

What 0.15.0 changed, according to nlnet labs, the author, was DNSSEC related.
What about shutting down DNSSEC entirely for a while ?

scottlindner · Dec 4, 2022, 1:45 PM

This post is deleted!

scottlindner · Dec 4, 2022, 4:43 PM

@gertjan said in DNS Resolver needs a constant reboot to work:

@scottlindner

This is the thread : Slow DNS after 22.0
It talks about 'buffers' and IPv6.

Btw : a thread like that, I hate it.
I've been trying a lot to create the same unbound behaviour.
I'm using an Intel Atom, a Netgate SG 4100 device, I do use IPv6 on my LANs, not that I'm really needing it.
I'm using pfBlockerng-devel, with a "restart reload feeds ones a week" as I don't want unbound to get restarted every 60 minutes or so, because some one somewhere added one DNSBL on some list I use.

The thing is : if we use the same hardware (a Netgate device), the some software, then what is different ?
Our LAN : the cables, switches and devices.

I guess, my unbound knows I'm watching him.
See here what I use so I can see what happens when. That's my unbound. It restart a lot, because, while I'm writing here, I do try stuff before posting, my unbound gets restarted.
But when I'm not poking in my pfSense, you can see unbound runs for days or weeks without a restart.
Of course, if unbound stops handling DNS for my LANs, credit card machines start to fail .... and then all hell breaks loose. As money stops coming in.

An there is of course Unbound 1.15.0 released and this version was replaced by 0.15.1 , 0.15.2 etc.
These sub versions came out because of 'issues'. pfSense doesn't allow us (easily) to try all these versions, its "whatever they chose" up until the next pfSense release comes out.

What 0.15.0 changed, according to nlnet labs, the author, was DNSSEC related.
What about shutting down DNSSEC entirely for a while ?

I had already disabled DNSSEC from a while ago when I was trying to bang on this issue before.

Is there a way to reset DNS Resolver settings to default? I couldn't find anything. I just want to be sure I haven't done something unintentional as I have been trying to get my Internet to work without having to man the pfSense console constantly.

scottlindner · Dec 4, 2022, 4:45 PM

Crap. I tried upgrading to the latest experimental just to try to force my way out of this problem and now upgrading is jacked too. It failed with meaningless "unable to..." errors and now the Update system fails to check for updates. Guess I gotta rebuild this thing... again^3.

If I save off the XML is there an effective way to hack the XML for a clean install but to make sure Unbound is set to defaults? I know I can just manually rebuild everything and maybe that's what I'll do. Getting the VLANs back to working is the pivotal part for me because until that happens I gotta use a laptop connected to the Netgate.

Gertjan · Dec 5, 2022, 8:17 AM

@scottlindner said in DNS Resolver needs a constant reboot to work:

Crap. I tried upgrading to the latest experimental

You went from 22.05 to something new ?
I wouldn't dare doing so.

Read some posts from here : Home > pfSense Software > Development should make you think otherwise.

@scottlindner said in DNS Resolver needs a constant reboot to work:

Guess I gotta rebuild this thing... again

You have a good backup up config file when you left 22.05 ? That's the one you need.
If not, take one from here : /cf/conf/backup : look at the date / time stamp, and use the one you where using with 22.05.

Installing a fresh 22.05, assign a minimal LAN + WAN + import xml + reboot and you'll be back to square one.

edit : extra info :

https://forum.netgate.com/topic/174248/need-help-troubleshooting-dns-after-upgrade-to-22-05

scottlindner · Dec 9, 2022, 7:56 PM

@gertjan said in DNS Resolver needs a constant reboot to work:

@scottlindner said in DNS Resolver needs a constant reboot to work:
Crap. I tried upgrading to the latest experimental 
You went from 22.05 to something new ?
I wouldn't dare doing so.

Read some posts from here : Home > pfSense Software > Development should make you think otherwise.

@scottlindner said in DNS Resolver needs a constant reboot to work:

Guess I gotta rebuild this thing... again

You have a good backup up config file when you left 22.05 ? That's the one you need.
If not, take one from here : /cf/conf/backup : look at the date / time stamp, and use the one you where using with 22.05.

Installing a fresh 22.05, assign a minimal LAN + WAN + import xml + reboot and you'll be back to square one.

edit : extra info :

https://forum.netgate.com/topic/174248/need-help-troubleshooting-dns-after-upgrade-to-22-05

I just reinstalled and restored the latest config. The update check is now working. I'm kinda hopeful my DNS Resolver issues just magically go away. I did save off a copy of the config XML in the default fresh installation state so I can hack an XML file to restore certain features to default. This being the key that I was looking for in this particular thread:

	<unbound>
		<enable></enable>
		<dnssec></dnssec>
		<active_interface></active_interface>
		<outgoing_interface></outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
	</unbound>

This is the unbound config that I restored.

	<unbound>
		<enable></enable>
		<active_interface>all</active_interface>
		<outgoing_interface>all</outgoing_interface>
		<custom_options></custom_options>
		<hideidentity></hideidentity>
		<hideversion></hideversion>
		<dnssecstripped></dnssecstripped>
		<port></port>
		<sslcertref>589929552f3cb</sslcertref>
		<regdhcpstatic></regdhcpstatic>
		<system_domain_local_zone_type>transparent</system_domain_local_zone_type>
		<tlsport></tlsport>
	</unbound>

Would any of those differences cause issues? Seems pretty trivial and more like skeleton config rather than influential.

scottlindner · Dec 9, 2022, 8:05 PM

@gertjan

One reason I love running my Netgate. Never would have gotten this with a traditional residential router.

scottlindner · Dec 14, 2022, 12:47 PM

This is interesting. After reinstalling pfSense and restoring my configuration things seem much more stable. I know it's still early to know for sure. I even shut off the reboot and unbound restart cron jobs and things are good so far. My previous pfSense installation was a lot of upgrades so I'm wondering if some configuration got left behind that a clean install took care of that doesn't show up in the XML but does on the device OS.

In the future when there is an update available for my Netgate, I'm going to do a clean install and config restore rather than upgrade.

t__2 · Jan 14, 2023, 2:37 PM

I have the exact same problem here with a 5100. Pfsense stops providing DNS services to everything on our internal network. unbound has not stopped. It shows it is still running. I can do a ping from the Diagnostic menu and it resolves the IP just fine and the ping works but only from inside the pfsense gui. I have not tried it when ssh'd into the 5100 yet.

Fortunately for us it only does this every few weeks. So when it does I log into the gui and restart unbound and everything works again. I am also running OpenVPN. I might be retiring OpenVPN soon for Wiregaurd. We will see if that changes anything.

scottlindner · Jan 14, 2023, 2:37 PM

@t__2 said in DNS Resolver needs a constant reboot to work:

I have the exact same problem here with a 5100. Pfsense stops providing DNS services to everything on our internal network. unbound has not stopped. It shows it is still running. I can do a ping from the Diagnostic menu and it resolves the IP just fine and the ping works but only from inside the pfsense gui. I have not tried it when ssh'd into the 5100 yet.

Mine seems to be stable after a fresh install of pfSense. I saved off the config, installed fresh, loaded config and I seem to be fine now. Although I'm not going to say that with 100% confidence because sometimes it takes months.

I did have another similar issue but restarting Unbound didn't fix it but a reboot did. That happened twice so I set a Cron to reboot pfSense once a week and now I seem great. Again, waiting for long term to confirm.

t__2 · May 12, 2023, 7:03 PM

@scottlindner Switched to wireguard removed OpenVpn for pfsense but still had the problem. Upgrade to 23.01 on 2023-04-17 and have not had to reboot unbound since. Maybe they fixed it?

scottlindner · May 12, 2023, 9:51 PM

@t__2
For me it wasn't the upgrade so much as a clean install after the upgrade. Has been totally fine since the flash install.