DNS resolver issues when accessing MS services on Wifi only?

[kasanoff]

Hi all,
PFSense v24.03 running on Netgate 6100
Have DNS servers set to Google's and using DNS Resolver with Forwarding Mode enabled
Using pfBlockerNG

We use MS365 and I've discovered a number of users on our network are not able to reach MS services (e.g. send/receive email, browse to office.com) while connected to our network. It seems to be affecting WiFi users only and primarily mobile device users (e.g. I am still able to send/receive email on my own laptop, but not on my mobile device, both connected to WiFi) However, there are some laptop users on WiFi that are also affected.

I've narrowed it down to the DNS Resolver as when I disable this, it all starts working again for our users. However, we have some Host Overrides defined (locally hosted PBX) so I really want to figure out what going on with the DNS Resolver. If I manually set Google's DNS for a user on their mobile device/PC this also works, but I don't really want to do this either for the same reasons above.

Some more details:
System -> General Setup
DNS Servers (Google's)
DNS Server Override = unticked
DNS Resolution Behavior = Use Local DNS, fall back to remote DNS

Services -> DNS Resolver -> General Settings
DNS Resolver = ticked
DNSSEC = unticked
DNS Query Forwarding = ticked
Use SSL/TLS for outgoing DNS = ticked

I thought pfBlocker may have been causing issues but I have since disabled the DNSBL service and ultimately pfBlocker entirely and there is no change.

DNSlookup for "office.com" in pfSense - resolves correctly
NSlookup on a user's PC (wired ethernet) (using pfSense as DNS Server) - resolves correctly
NSlookup on a user's laptop (WiFi connected, using pfSense as DNS Server) - DNS request times out
NSlookup on same user's laptop (WiFi connected, specifying 8.8.8.8 DNS server) - DNS request times out
NSlookup on same user's laptop (5G tether) - resolves correctly

Not really sure where to go from here - any advice/suggestions would be appreciated

Cheers!

[Gertjan]

@kasanoff said in DNS resolver issues when accessing MS services on Wifi only?:

We use MS365 and I've discovered a number of users on our network are not able to reach MS services

Leave the computer, put on your TV, and watch.
AFAIK, Microsoft has some issues lately, and the impact is world wide.
"Crowdstrike"

@kasanoff said in DNS resolver issues when accessing MS services on Wifi only?:

Forwarding Mode enabled

Thanks.
I'll put that on my big "don't forward" list

[kasanoff]

@Gertjan said in DNS resolver issues when accessing MS services on Wifi only?:

Leave the computer, put on your TV, and watch.
AFAIK, Microsoft has some issues lately, and the impact is world wide.
"Crowdstrike"

Ah yeah - it is coincidental that I've posted at the same time as this global outage so if it suddenly resolves itself when this Crowdstrike stuff blows over I'll be thrilled lol!

But I have been dealing with this this DNS issue for days/weeks now so I'm not holding my breath

[Gertjan]

@kasanoff

You are aware that Google (8.8.8.8) is a resolver, and it uses these (and only these) ?

The resolver, unbound, that comes with pfSense also uses the same these. As why getting info from a third party if you can get it from the source ?
If any of them goes down, no issue, but this actually never happens.

So, what about applying 'Plan A' : use the revolver's default settings and call it a day ?

These days, if something goes wrong with Google, half the planet looses the Internet (just the DNS actually). Or, the 'Internet' doesn't 'need' Google, as it worked great decades before Google was invented.

[Uglybrian]

I would start by going over your settings on your WiFi access points. Where are they getting their DNS from?

[johnpoz]

You understand that there is/was a major outage due to crowdstrike right?

Effected lots of stuff MS being one of them.

[SteveITS]

I'm told news reports are reporting it as a Microsoft outage, but it is a Crowdstrike definition problem, that only happens to affect Windows PCs and servers (BSOD).

[johnpoz]

@SteveITS sure MS runs some of their own product ;) but lots of other companies running windows as well ;)

Not really a MS outage only, but yeah the crowdstrike is the cause.

Windows/System32/Drivers/CrowdStrike/C-00000291*.sys

[kasanoff]

After a bunch of testing today, I ended up checking traffic using pftop and compared entries when accessing MS services vs everything else (probably should have done this earlier)

Accessing other services - I see traffic destined for port 53, as expected
Accessing MS services - I see traffic destined for port 12000 (?)

Found another branded AP lying around, configured and connected to it and tried accessing MS services - I see a bunch of traffic to port 53 this time, no port 12000 to be found

NFI why the port is being changed. As a temp workaround I've translated port 12000 traffic back to port 53 and everything's working as expected. Now I'm chasing Netgear to ask W-T-F!

Cheers!