DNS Headaches Since Switching to PFSense

Ashkaan

@johnpoz because these wireless devices work perfectly with other firewalls.

johnpoz

@ashkaan Well that logic flawless <rolleyes>

So again I will state do a directed query.. Does it fail? Did pfsense even see the query.

Simple enough to do - grab say hurricane electric network phone app, allows to do a directed query to a specific dns.. Or your other fav tool for your wireless device that allows you to do a directed query.

https://networktools.he.net/

Now sniff on pfsense when your doing a query - does pfsense even see it?

example..

As you can see my client doesn't report answer - but can see that pfsense saw the query and did answer with nx..

If problem with pfsense (dns on pfsense) then you would see the query from your client in the sniff - but no answer at all..

Lets say that was the case - how would reconnecting the client to wifi fix that? Let me think - oh yeah it wouldn't have anything to do with it. ;)

Gertjan

I guess it's time for :

grep 'stop' /var/log/resolver.log

Do you see one are two stops (and starts) a day - or less ? : that's ok.
You could look at the /var/log/resolver.log file, and check how long it takes between a 'stop' and a 'start', as that will be the time that DNS queries are not answered.
At that moment, only DNS resolving doesn't work, which is just a small subset of the entire Internet access experience, but for some reason it makes people think that the connection is 'out'.

Or look here, as it is the same info : Status > System Logs > System > DNS Resolver

As Jonhpoz already proposed : test this : go to Services > DNS Resolver > General Settings and un check "DHCP Registration", Save and Apply.
Do the grep test again a couple of hours / days later. You should notice that unbound restarts less often.

Ashkaan

@johnpoz said in DNS Headaches Since Switching to PFSense:

@ashkaan Well that logic flawless <rolleyes>
So again I will state do a directed query.. Does it fail? Did pfsense even see the query.

I cannot test the address because I do not know the address that the random app that fails is trying to reach. That's why I suggested that I search a log to try to find it, but I don't know how to do that or if it's even possible.

Ashkaan

@gertjan I get the following when trying your command:

alt text

When I look through the GUI, I see a bunch of stops. Are you thinking that I'm running into issues with my devices only precisely when DNS is restarted? It seems like really tiny and infrequent windows for this to be going on.

alt text

Gertjan

@ashkaan

For admin task you need the default admin account.
pfSense is a router firewall, no need to create multiple user accounts - the 'admin' account is needed for most if not all interactions.

Ok, your close.
With every 'stopped' there is also a "start". The time between them is the time the network has no DNS available.

This :

is good news : no unbound restart for several days. So it's up and running.

If the LAN interfaces do not block DNS traffic; then there shouldn't be any DNS issues. No issues that are pfSense related..

Don't.
As you don't need them, except if you signed some sort of contract with them.
Unbound is a resolver - 8.8.8.8 is a resolver - 1.1.1.1 is a resolver. A resolver doesn't need a resolver to resolve. A resolvers uses the root DNS servers to work.

Ashkaan

@gertjan Thank you so much for the helpful reply. I have removed the external resolvers as they were unnecessary.

The challenge that I'm having is that everything works perfectly with other firewalls. It's only when I have PFSense running (again, as opposed to EdgeRouter or Sonicwall) that I randomly have the issue. I never had the issue with those other platforms.

I assumed that it was DNS related because it almost looks like the app can't resolve or isn't connecting to the internet, BUT I have a constant ping (to Google) running on a server that has no gaps. There's no internet outage here.

Does anyone have any other leads for me to chase down?

johnpoz

@ashkaan And what app is this?

So what IP does your phone get.. Why don't you sniff (diag menu packet capture) and actually see what is going on.

So this title should be changed - because you really have no clue to what the issue is.. You don't even know where your app is trying to go? Or what its trying to do..

If you were having dns problems - that should present itself as lots of stuff not working, or atleast this one thing you know not working on every device using pfsense for dns, etc. etc.

Out of the box pfsense does no filtering outbound, and does not block any dns.. Have you changed this default? Are you using pfblocker? Are you running IPS/IDS?

Turning wifi on and off on the phone would do ZERO to pfsense, sure and the hell wouldn't fix a dns issue on pfsense, etc. So if turning on wifi on your phone and back on fixes your issues - that screams something wrong with your phone or your wifi..

So you have no wired devices? Do they have problems?

Ashkaan

@johnpoz It's many apps from games to crypto apps to Nest. It's not a single app, but the phones and iPads just stop working for a moment. I switch to cell service (off the network) and they work perfectly. The server never misses a ping. This doesn't happen on other firewalls

It would be impossible for me to guess when it will happen to run a packet capture at that exact time. It also only happens a few times a week, sometimes once a day.

Feel free to change the title. From my experience, this looks like a DNS issue and I don't have evidence to the contrary at the moment.

This is a VERY default setup. I don't like complicating things so I have not messed with anything.

Yes, I understand that turning off WiFi would not DO anything to PFSense, but it proves that there's something wrong with the network. Again, the WiFi works great with other firewalls, so we know the PFSense is the common denominator here.

My only wired device is my server and I don't browse the net often enough on it to tell. I can tell you that it never misses a ping. I suspect (only guessing) that because all other devices that I regularly use have the issue that it also has the issue since I ruled out the WiFi being the issue.

johnpoz

@ashkaan said in DNS Headaches Since Switching to PFSense:

It also only happens a few times a week, sometimes once a day.

Well that points to the dns restarting issue we have brought up a couple of times already..

unbound goes off, you go to switch your wifi off and on and by time you come back its restarted and you think turning off wifi and back on fixed it.

Turn off registration of your dhcp clients.. Watch how often unbound is restarting - and the next time you have the issue, look in your logs - did unbound just happen to restart?

Next time it happens - before you go flipping yoru wifi on and off - do the directed query I gave as example to something, www.google.com - or something you haven't gone to in a while so your sure its not cached, etc.

If you want to get to the root of the problem your going to have to do something more than flipping yoru wifi on and off..

Ashkaan

@johnpoz Yes, thank you for reminding me. Ok, I'll test that now. Just to confirm, it should look like this to prevent further DNS reboots?

alt text

johnpoz

@ashkaan yes no registering dhcp clients should keep unbound from restarting every time a dhcp client renews or gets an IP.. You can check your unbound uptime with this.

[22.01-RELEASE][admin@sg4860.local.lan]/: unbound-control -c /var/unbound/unbound.conf status
version: 1.13.2
verbosity: 1
threads: 4
modules: 2 [ validator iterator ]
uptime: 334795 seconds
options: control(ssl)
unbound (pid 32247) is running...
[22.01-RELEASE][admin@sg4860.local.lan]/:

So mine has been up for 334795 seconds or 93 hours.. I have seen uptimes much much longer than that - but I tend to restart mine quite a bit testing stuff for users and their threads.

You can register static if you want, these would be like your host overrides..

Ashkaan

@johnpoz Ok, thank you so much! I'll keep everyone posted how this goes.

Gertjan

@ashkaan
We know already how it will go ;)

Just check your resolver log file in the GUI, and look for the 'stop', where unbound logs that is about to stop - to be restarted right away.
The frequency will be a lot lesser, maybe one a day or even once a week.
Btw : saving DNS or DNS related settings will restart unbound also.

If this was your issue, it will be gone.
unbound could still be restarted, as there are many more reasons why it would restart, but these are (should be) less frequent.
And remember : even if unbound restart, DNS will be stopped for just a couple of seconds.
That should be a non-event for devices and users.

Things start to be noticeable when the network changes often :
The admin rips out LAN or WAN cables : this creates a network OS event, and 'attached' processes that listen on these interfaces will restart.
A NIC or cable could be bad : it loses the carrier every x seconds, then it works fine for hours, the result will be the same. Random network access.
Or : a switch is hidden in the attic has a bad power supply : it resets very often, all ports go down - and this will create the same event as mentioned above.

And the most important one :
As people can't smell, see or hear radio signals, they are considered to be "working perfectly well".
Or, in reality, the contrary is most often the case.
Just last weekend a showed a friend why it's iPhone was horribly slow, or, he wanted to show me his latest unifi-ap-6-lite (long range !) that did not from well a 30 meters (we use meters here, as our feet do not have a fixed size).
I showed him that his iPhone negotiated a less then 2Mbits/sec connection. Now my friiend is member of the club : "Ok for the high speed, but transmit/receive range size really became very small"
The radio frequency spectrum is often a mess. radio communication is a mess. People just don't stop adding 'connected' devices, and because they can't "see" what these devices are doing they are presumed to "will work just fine". What happens to constructive pessimism ?
Depending on what you said when you were at school, they would have gave you a a spectrum analyser (instead of a adjustable wrench), and you would have seen the invisible : lets look at the classic FM band, the band where the TV was - here in Europe it was called the VHF & UHF band. VHF and the top of UHF has been recycled now. TV is now compressed the digital way, 5 or 6 channels a carrier, and the space that was freed up is now called : 5G (even more mess). Go a bit higher and you enter the micro wave frequency. And the Boeing 737-x00 alti meter, a device used to stabilize the approach by measuring the distance between the plane and ground.
I saw what was there in the end of the '70. No wifi bleutooh and only 3 national TV channels back then.

Be assured, I'm not against the usage of Wifi (I'm not Swedish ) but it is an important "does not work well" factor.

Same thing for our ISP upstream connection : people tend to think, "I pay, so it works".
But noop, ISPs are not all identical but they do all the same thing : they give you what they have. They all sell their bandwidth "many times". So, many clients have the impression that they "can't get it all". Or worse, (small) DNS packets get dropped.
Or, and this should happen a lot : for the 8.8.8.8 lovers : what happens when 8.8.8.8 gets over run ? Well ? people drop in here, and say there are issues with pfSense. The ones who know a little bit more would say "DNS error" The even smarter ones would say ... nothing. They'll suddenly ask themselves : why should my resolver forward to a resolver ? resolvers like 8.8.8.8 use the 13 main root internet servers. So pfSense could do the same ?!!!

[ ranting mode = stop]

Ashkaan

@gertjan Haha! Well, thank you very much for the insights. I’ll keep a look out and let you know how it goes.

Ashkaan

@ashkaan Still happening. It happened at 8a yesterday and again at 832a this morning.

Unfortunately, there's nothing in the DNS logs about this around those times. I'm totally baffled. Where should I look next?

Again, the symptom is that apps and websites in Safari will just hang. As soon as I switch to Cell, they load. My server hasn't dropped a ping and I've since unplugged my desktop and put on WiFi and started a ping and that's solid too.

johnpoz

@ashkaan said in DNS Headaches Since Switching to PFSense:

there's nothing in the DNS logs about this around those times

So your unbound has been up for how long, once you had the issue. Did you try a directed dns query?

What specific website did not load - you need to have a specific thing that is not working if you want to troubleshoot..

What exactly are you pinging? By IP or by name? Are you pinging the specific site that is not working?

Are you forwarding for dns, or did you change to resolving? Is your client using doh for dns? Are you using IPv6 or just IPv4.. Without specific it is impossible to try and figure out what exactly is going on that your seeing the issue your seeing.

Ashkaan

@johnpoz I was in a work emergency so I didn't have time to try a direct query, but it was google.com of all sites! Google wouldn't load in Safari. On cell, it loaded immediately. The WiFi desktop didn't lose a ping to 8.8.8.8.

It's been up since midnight of the prior day, so it wasn't stopped at the issue times.

I was using Resolving, then I tested Forwarding for a couple of weeks, then I went back to Resolving about a week ago. No DoH. Only IPv4.

thiasaef

@johnpoz, with https://redmine.pfsense.org/issues/12613 still not being fixed I would honestly be more surprised if it did work in a typical home setup then when it did not ... and while we're at it - please fix it properly!

johnpoz

@thiasaef said in DNS Headaches Since Switching to PFSense:

https://redmine.pfsense.org/issues/12613

He would be able to tell if that was what he was running into - by simple query for something. And he is never restarting dns - so how would that be the issue. Just flipping a client wifi on off wouldn't restart unbound, etc.

I don't see how there is enough info to say what is going on one way or the other.

What I can say is I have not run into any issues, nor that specific issue at all.. I don't use a vpn for vpn traffic, nor have unbound bound to the test vpn connection I leave up. I don't have multiple interfaces that could be going up or down.

And I bind unbound only to loopback anyway, and just let it nat outbound, etc. So even if my wan went offline I should be fine, but my wan connection is pretty rock solid as well.

What needs to happen to move forward here is some actual specifics of what is not working, and when. I get it you don't always have time to troubleshoot when something goes wrong, and you just want it back up now.. But if want to get to the bottom of the actual problem - going to need to actually get some details when it happens..