DNS Headaches Since Switching to PFSense

Ashkaan

ROFL.. I'm just reading that it specifically says not to use that. OMG.. is THIS is the reason?

What should I use? I don't want to use a public address.

SteveITS

@ashkaan As it says blah.local.lan or .mylocal or .ashkaan or anything unlikely to be used. If you had your own domain, lan.example.com.

Devices can also use DoH for DNS.

Ashkaan

@steveits Is this why some apps and websites don't resolve?

johnpoz

@ashkaan said in DNS Headaches Since Switching to PFSense:

Is this why some apps and websites don't resolve?

No while .local shouldn't be used, it wouldn't cause other fqdn to fail. But it could be causing you local issues?

The current recommend local domain to use would be something.home.arpa

https://datatracker.ietf.org/doc/html/rfc8375
Special-Use Domain 'home.arpa.'

I have used unbound on pfsense, since before it was built in, and just a package.. While have extensive bind experience and have used that professionally since really it was a thing, and use to use it for my home dns.. Unbound is easy to use, it is very robust and more than capable of providing a recursive name services for most uses.. It is not really meant to authoritative, but more than enough for most uses in a small business or home setup. It doesn't have all the bells and whistles that bind does..

The only real problem I have seen, and most likely the problem you might be having when it just looks like some stuff doesn't resolve is registering dhcp - which causes unbound to restart, and when also used with pfblocker the restart can take longer than normal.

If you have an issue with something specific not resolving.. Troubleshoot that specific fqdn - if your resolving, you could have an issue talking to roots or gtld servers, or just not able to talk to the actual authoritative domain. Or maybe its high latency issue to the authoritative NS for that specific domain?

dig and +trace is very valuable tool in troubleshooting such an issue. If your forwarding - and not getting an answer, that is on where your forwarding to, or can not just talk to them?

When you have a problem next time - what is the specific fqdn? Do a directed query, use the dns lookup tool under diagnostics menu. Does it resolve? Use your fav tool, dig, host, nslookup to do some directed queries - does say unbound answer for something local, like pfsense fqdn? But not just this specific fqdn? etc..

But I would really first thing to check is if unbound is restarting - but if you were actually using dnsmasq (the forwarder) it doesn't have the dhcp restart issue that unbound has..

Ashkaan

@johnpoz Thank you.

I don't think that I was having any local issues with the ".local" domain, but I switched to ".local.lan" anyway.

The challenge with testing the fqdn is that I have no idea what it is. Randomly (like once a day), an app on my phone won't work (different apps). I need it for work, so I just go off of WiFi and it immediately works. There's no network outage because I started running pings in the background and witnessed it happening while the pings looked great.

Is there an easy way to see DNS requests that haven't been fulfilled?

Again, this happens with Unbound AND the DNS forwarder in case that helps.

johnpoz

@ashkaan who says its anything to do with pfsense - sounds like a wireless issue to me.. Do you have any issues with wired devices and dns?

Ashkaan

@johnpoz because these wireless devices work perfectly with other firewalls.

johnpoz

@ashkaan Well that logic flawless <rolleyes>

So again I will state do a directed query.. Does it fail? Did pfsense even see the query.

Simple enough to do - grab say hurricane electric network phone app, allows to do a directed query to a specific dns.. Or your other fav tool for your wireless device that allows you to do a directed query.

https://networktools.he.net/

Now sniff on pfsense when your doing a query - does pfsense even see it?

example..

As you can see my client doesn't report answer - but can see that pfsense saw the query and did answer with nx..

If problem with pfsense (dns on pfsense) then you would see the query from your client in the sniff - but no answer at all..

Lets say that was the case - how would reconnecting the client to wifi fix that? Let me think - oh yeah it wouldn't have anything to do with it. ;)

Gertjan

I guess it's time for :

grep 'stop' /var/log/resolver.log

Do you see one are two stops (and starts) a day - or less ? : that's ok.
You could look at the /var/log/resolver.log file, and check how long it takes between a 'stop' and a 'start', as that will be the time that DNS queries are not answered.
At that moment, only DNS resolving doesn't work, which is just a small subset of the entire Internet access experience, but for some reason it makes people think that the connection is 'out'.

Or look here, as it is the same info : Status > System Logs > System > DNS Resolver

As Jonhpoz already proposed : test this : go to Services > DNS Resolver > General Settings and un check "DHCP Registration", Save and Apply.
Do the grep test again a couple of hours / days later. You should notice that unbound restarts less often.

Ashkaan

@johnpoz said in DNS Headaches Since Switching to PFSense:

@ashkaan Well that logic flawless <rolleyes>
So again I will state do a directed query.. Does it fail? Did pfsense even see the query.

I cannot test the address because I do not know the address that the random app that fails is trying to reach. That's why I suggested that I search a log to try to find it, but I don't know how to do that or if it's even possible.

Ashkaan

@gertjan I get the following when trying your command:

alt text

When I look through the GUI, I see a bunch of stops. Are you thinking that I'm running into issues with my devices only precisely when DNS is restarted? It seems like really tiny and infrequent windows for this to be going on.

alt text

Gertjan

@ashkaan

For admin task you need the default admin account.
pfSense is a router firewall, no need to create multiple user accounts - the 'admin' account is needed for most if not all interactions.

Ok, your close.
With every 'stopped' there is also a "start". The time between them is the time the network has no DNS available.

This :

is good news : no unbound restart for several days. So it's up and running.

If the LAN interfaces do not block DNS traffic; then there shouldn't be any DNS issues. No issues that are pfSense related..

Don't.
As you don't need them, except if you signed some sort of contract with them.
Unbound is a resolver - 8.8.8.8 is a resolver - 1.1.1.1 is a resolver. A resolver doesn't need a resolver to resolve. A resolvers uses the root DNS servers to work.

Ashkaan

@gertjan Thank you so much for the helpful reply. I have removed the external resolvers as they were unnecessary.

The challenge that I'm having is that everything works perfectly with other firewalls. It's only when I have PFSense running (again, as opposed to EdgeRouter or Sonicwall) that I randomly have the issue. I never had the issue with those other platforms.

I assumed that it was DNS related because it almost looks like the app can't resolve or isn't connecting to the internet, BUT I have a constant ping (to Google) running on a server that has no gaps. There's no internet outage here.

Does anyone have any other leads for me to chase down?

johnpoz

@ashkaan And what app is this?

So what IP does your phone get.. Why don't you sniff (diag menu packet capture) and actually see what is going on.

So this title should be changed - because you really have no clue to what the issue is.. You don't even know where your app is trying to go? Or what its trying to do..

If you were having dns problems - that should present itself as lots of stuff not working, or atleast this one thing you know not working on every device using pfsense for dns, etc. etc.

Out of the box pfsense does no filtering outbound, and does not block any dns.. Have you changed this default? Are you using pfblocker? Are you running IPS/IDS?

Turning wifi on and off on the phone would do ZERO to pfsense, sure and the hell wouldn't fix a dns issue on pfsense, etc. So if turning on wifi on your phone and back on fixes your issues - that screams something wrong with your phone or your wifi..

So you have no wired devices? Do they have problems?

Ashkaan

@johnpoz It's many apps from games to crypto apps to Nest. It's not a single app, but the phones and iPads just stop working for a moment. I switch to cell service (off the network) and they work perfectly. The server never misses a ping. This doesn't happen on other firewalls

It would be impossible for me to guess when it will happen to run a packet capture at that exact time. It also only happens a few times a week, sometimes once a day.

Feel free to change the title. From my experience, this looks like a DNS issue and I don't have evidence to the contrary at the moment.

This is a VERY default setup. I don't like complicating things so I have not messed with anything.

Yes, I understand that turning off WiFi would not DO anything to PFSense, but it proves that there's something wrong with the network. Again, the WiFi works great with other firewalls, so we know the PFSense is the common denominator here.

My only wired device is my server and I don't browse the net often enough on it to tell. I can tell you that it never misses a ping. I suspect (only guessing) that because all other devices that I regularly use have the issue that it also has the issue since I ruled out the WiFi being the issue.

johnpoz

@ashkaan said in DNS Headaches Since Switching to PFSense:

It also only happens a few times a week, sometimes once a day.

Well that points to the dns restarting issue we have brought up a couple of times already..

unbound goes off, you go to switch your wifi off and on and by time you come back its restarted and you think turning off wifi and back on fixed it.

Turn off registration of your dhcp clients.. Watch how often unbound is restarting - and the next time you have the issue, look in your logs - did unbound just happen to restart?

Next time it happens - before you go flipping yoru wifi on and off - do the directed query I gave as example to something, www.google.com - or something you haven't gone to in a while so your sure its not cached, etc.

If you want to get to the root of the problem your going to have to do something more than flipping yoru wifi on and off..

Ashkaan

@johnpoz Yes, thank you for reminding me. Ok, I'll test that now. Just to confirm, it should look like this to prevent further DNS reboots?

alt text

johnpoz

@ashkaan yes no registering dhcp clients should keep unbound from restarting every time a dhcp client renews or gets an IP.. You can check your unbound uptime with this.

[22.01-RELEASE][admin@sg4860.local.lan]/: unbound-control -c /var/unbound/unbound.conf status
version: 1.13.2
verbosity: 1
threads: 4
modules: 2 [ validator iterator ]
uptime: 334795 seconds
options: control(ssl)
unbound (pid 32247) is running...
[22.01-RELEASE][admin@sg4860.local.lan]/:

So mine has been up for 334795 seconds or 93 hours.. I have seen uptimes much much longer than that - but I tend to restart mine quite a bit testing stuff for users and their threads.

You can register static if you want, these would be like your host overrides..

Ashkaan

@johnpoz Ok, thank you so much! I'll keep everyone posted how this goes.

Gertjan

@ashkaan
We know already how it will go ;)

Just check your resolver log file in the GUI, and look for the 'stop', where unbound logs that is about to stop - to be restarted right away.
The frequency will be a lot lesser, maybe one a day or even once a week.
Btw : saving DNS or DNS related settings will restart unbound also.

If this was your issue, it will be gone.
unbound could still be restarted, as there are many more reasons why it would restart, but these are (should be) less frequent.
And remember : even if unbound restart, DNS will be stopped for just a couple of seconds.
That should be a non-event for devices and users.

Things start to be noticeable when the network changes often :
The admin rips out LAN or WAN cables : this creates a network OS event, and 'attached' processes that listen on these interfaces will restart.
A NIC or cable could be bad : it loses the carrier every x seconds, then it works fine for hours, the result will be the same. Random network access.
Or : a switch is hidden in the attic has a bad power supply : it resets very often, all ports go down - and this will create the same event as mentioned above.

And the most important one :
As people can't smell, see or hear radio signals, they are considered to be "working perfectly well".
Or, in reality, the contrary is most often the case.
Just last weekend a showed a friend why it's iPhone was horribly slow, or, he wanted to show me his latest unifi-ap-6-lite (long range !) that did not from well a 30 meters (we use meters here, as our feet do not have a fixed size).
I showed him that his iPhone negotiated a less then 2Mbits/sec connection. Now my friiend is member of the club : "Ok for the high speed, but transmit/receive range size really became very small"
The radio frequency spectrum is often a mess. radio communication is a mess. People just don't stop adding 'connected' devices, and because they can't "see" what these devices are doing they are presumed to "will work just fine". What happens to constructive pessimism ?
Depending on what you said when you were at school, they would have gave you a a spectrum analyser (instead of a adjustable wrench), and you would have seen the invisible : lets look at the classic FM band, the band where the TV was - here in Europe it was called the VHF & UHF band. VHF and the top of UHF has been recycled now. TV is now compressed the digital way, 5 or 6 channels a carrier, and the space that was freed up is now called : 5G (even more mess). Go a bit higher and you enter the micro wave frequency. And the Boeing 737-x00 alti meter, a device used to stabilize the approach by measuring the distance between the plane and ground.
I saw what was there in the end of the '70. No wifi bleutooh and only 3 national TV channels back then.

Be assured, I'm not against the usage of Wifi (I'm not Swedish ) but it is an important "does not work well" factor.

Same thing for our ISP upstream connection : people tend to think, "I pay, so it works".
But noop, ISPs are not all identical but they do all the same thing : they give you what they have. They all sell their bandwidth "many times". So, many clients have the impression that they "can't get it all". Or worse, (small) DNS packets get dropped.
Or, and this should happen a lot : for the 8.8.8.8 lovers : what happens when 8.8.8.8 gets over run ? Well ? people drop in here, and say there are issues with pfSense. The ones who know a little bit more would say "DNS error" The even smarter ones would say ... nothing. They'll suddenly ask themselves : why should my resolver forward to a resolver ? resolvers like 8.8.8.8 use the 13 main root internet servers. So pfSense could do the same ?!!!

[ ranting mode = stop]