ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden
-
Hello, here I am again with a strange issue that I cannot resolve on my own. Basically, all of a sudden, multiple websites (including reddit.com) cannot be resolved... This is not the first time I have this issue and strangely enough, things normally come back to normal if I simply wait...
I tried deactivating pretty much everything relating to traffic control in pfsense (snort, pfblocker, DNSBL, firewall) to no avail. The only solution (other than just waiting), is to use a VPN.
Without connecting to VPN, the error I get in Chrome/Brave is :
This site can’t be reached www.reddit.com’s DNS address could not be found. Diagnosing the problem. DNS_PROBE_POSSIBLEThe "host" CLI command also cant resolve the domain's IP:
Host reddit.com not found: 3(NXDOMAIN)pfsense's DNS servers are (under General Setup) Quad9's servers with 9.9.9.9 & 149.112.112.112. I also use unbound (with python module for DNSBL) in FW mode and the following custom options:
server: forward-zone: name: "." forward-ssl-upstream: yes forward-addr: 9.9.9.9@853 forward-addr: 149.112.112.112@853How do I diagnose what's blocking the DNS resolutions?
-
@pftdm007 If you are forwarding.. Then if you can not resolve something its on who you forward too.. But that is not really the way you would forward, and your forwarding to TLS?
You didn't put those setting in the custom option box are you? You pulled those settings out of unbound conf file, and setup forwarding in gui?
Also if you forward - you should make sure you turn off dnssec - Or I am telling you your going to have a bad day.. Even quad9 on their faq says so..
https://docs.quad9.net/Quad9_For_Organizations/DNS_Forwarder_Best_Practices/
Disable DNSSEC Validation
Since Quad9 already performs DNSSEC validation, DNSSEC being enabled in the forwarder will cause a duplication of the DNSSEC process, significantly reducing performance and potentially causing false BOGUS responses.
also per quad9, did you setup the fqdn for forwarding to their dot ? dns.quad9.net - part of dot and doh is validation of who your talking too.. So if your going to use dot, you really should set the fqdn to validate
-
@johnpoz I assume these DNS issues are probably not related to misconfiguration(s) since most of the time things just work well... If it was a misconfiguration (and a serious one), either things would not work at all or rarely work well... Am I wrong to assume so?
This DNS resolution issue is intermittent. I've seen it a handful of times in the last year or so....
I have a hard time to understand your 1st sentence. Are you saying that because I am FW, the "place" I am FW is the culprit? Yes apparently I FW to TLS based on the custom settings of unbound. Not sure where I got this in the past, maybe from a Quad9 tutorial or setup blog?
DNSSEC has always been OFF
pfsense DNS settings are
@johnpoz said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
also per quad9, did you setup the fqdn for forwarding to their dot ?
I believe so (from the General setup)...
Some of unbound's setup
-
-
@pftdm007 that is not the way to forward.. So does that work sometimes not sure...
https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html#configuring-dns-over-tls
But again if you forward and where you forward sends you back a NX... Its on them, not on the client..
If you ask billy hey you have freds phone number, and he tells you no.. Who at fault? Did you ask for it wrong, did you really mean to ask for freddys number? And hey only knows him by freddy?
I agree if there was some blatent wrong you would think you would have either just horrible failure rate, or nothing would work.. But how your doing it is not the recommended way, and where in there can you validate quad9.. using dns.quad9.net
If you were just forwarding I could tell multiple methods to see if they are messing with your dns, but if your going to encrypt it and they send you back NX, and you trust them, but you didn't validate them - how do we know exactly was going on..
Host reddit.com not found: 3(NXDOMAIN)
NX would come who you forwarded too.. Or he didn't know how to forward?? And telling you hey I don't have that record in my local data..
pfsense config page and everything seems OK to me...
You went over that - but your not doing it that way are you.. Where on that page does it say anything about putting stuff in the custom option box?
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
@johnpoz said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
and where in there can you validate quad9.. using dns.quad9.net
Well, first of all in the DNS resolver logs I see a lot of
Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 9.9.9.9#853 Mar 4 22:03:28 unbound 27738 [27738:2] info: reply from <.> 149.112.112.112#853 Mar 4 22:03:28 unbound 27738 [27738:2] debug: sending to target: <.> 149.112.112.112#853But I also see some of those:
Mar 4 22:01:40 unbound 27738 [27738:2] debug: tcp error for address 9.9.9.9 port 853Secondly, in the states (as instructed on pfsense's config page), I see a lot of connections to Quad9's servers over port 853:
WAN tcp XXX.XXX.XXX.XXX:62219 -> 149.112.112.112:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:49236 -> 9.9.9.9:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:29203 -> 149.112.112.112:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:21627 -> 9.9.9.9:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:63379 -> 9.9.9.9:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:48545 -> 9.9.9.9:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 B WAN tcp XXX.XXX.XXX.XXX:16872 -> 149.112.112.112:853 SYN_SENT:CLOSED 1 / 0 60 B / 0 BSince a picture is worth a 1000's words...
-
@johnpoz said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
@pftdm007 that is not the way to forward.. So does that work sometimes not sure...
https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html#configuring-dns-over-tls
Not the way ? I'm curious, as the settings I saw do look fine to me.
Like "configuring-dns-over-tls" explained.Btw : adding this in the custom block :
isn't needed anymore for many years now.
This :
@pftdm007 said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
Secondly, in the states (as instructed on pfsense's config page), I see a lot of connections to Quad9's servers over port 853:
could be important. IIRC : there was an identical forum thread some time ago .... 9.9.9.9 (actually : nobody) doesn't like to get chain gunned with 'very expensive' TLS connections. If they apply throttling then that would explain what happens. Normally, unbound shuld keep exiting TLS connections open for future usage .... again IIRC.
And another thing : if I had to push the number of DNS connections up for whatever reason, I would not 'risk' my own WAN "IP" connection, I would use a VPN ISP ^^
Now, if things go bad, the IP would get 'blacklisted" and I don't care : when I'm done, I throw it away ^^
And no you drop in, using the same VPN ISP, and you get my previous IP ...... you get the point ? ^^
9.9.9.9 knows of course you use a "VPN ISP IP", so maybe they throttle you right from the beginning.So : why forwarding ? True : a possible gain : DNS requests will get TLS protected.
On the down side :
Just one DNS end point.
TLS is xxxxx% times more resource expensive.
'They' know where you ....
One last thing :
@pftdm007 said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
Without connecting to VPN, the error I get in Chrome/Brave is :
Always double check your browsers for DNS usage !
A browser doesn't have to use the DNS of the OS of 'host device'. For example : it doesn't have to use the DNS IP the OS obtained by using the DHCP-client - normally this is 192.168.1.1, the pfSense LAN IP.
A browser could very well using 8.8.8.8 (over TLS, why not) using not port '53' but some other port.
This means that all browser DNS requests will completely bypass "pfSense", the resolver.
Most browser put in place such a scheme these days. This is because they "want to protect you".
The real reason is : they want to know where you go, as this info is worth $$. -
@Gertjan said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
So : why forwarding ? True : a possible gain : DNS requests will get TLS protected.
On the down side :
Just one DNS end point.
TLS is xxxxx% times more resource expensive.
'They' know where you ....You're saying no forwarding with TLS?
@Gertjan said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
Always double check your browsers for DNS usage !
Browsers are fine. They all throw pretty much the same error...
@Gertjan said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
The real reason is : they want to know where you go, as this info is worth $$.
Most technologies are nefarious to a certain extent nowadays...
-
Quad9 is totally fu***ing up with me. I tried a few other DNS's (should have tried this to begin with, d'oh) and using google's dns (which I refrain due to my absolute and unconditional lack of trust for google) everything is working normally.
So.... I believe that @Gertjan may be right with rate limiting of some sort. Question is why would they rate limit me when I am only a home user? Its not like I'm running servers and websites with TB's of traffic and billions of requests per day....
-
@pftdm007 said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
You're saying no forwarding with TLS?
Exact.
Really : try for yourself. Its like magic. Using the DNS as it was meant to be used: getting the info 'from the source' without any intermediates.So, no I do not need to forward to anyone.
-
@pftdm007 said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
Question is why would they rate limit me
You as a person with your device : maybe not.
But .... there was (still is ?) an issue with unbound in the was recycling 'expensive' TLS connection very fast .... and forwarders don't like this at all.
Compare with another one 1.1.1.1 or 8.8.8.8 etc.
Or, as I said above : your IP is (was ?) 'suspect' as it is and stays a VPN ISP IP.Btw : when I was testing 9.9.9.9 I saw the same thing ....
-
@pftdm007 said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
Quad9 is totally fu***ing up with me. I tried a few other DNS's (should have tried this to begin with, d'oh) and using google's dns (which I refrain due to my absolute and unconditional lack of trust for google) everything is working normally.
So.... I believe that @Gertjan may be right with rate limiting of some sort. Question is why would they rate limit me when I am only a home user? Its not like I'm running servers and websites with TB's of traffic and billions of requests per day....
I have the same issue today with Quad9 which I have been using for a long time without issues.
So either they have changed something today, or UNBOUND has reached a limit or something in its code today that makes it act differently. -
@keyser said in ISP blocking or interfering with traffic? Cant resolve certain domains all of a sudden:
I have the same issue today with Quad9 which I have been using for a long time without issues.
So either they have changed something today, or UNBOUND has reached a limit or something in its code today that makes it act differently.Wonder if this might somehow be related to the massive Meta/Facebook/Google auth outage being reported worldwide (started around 10:00 AM EST this morning). Meta/Facebook/Instagram are massively impacted, but lots of reports of Google auth issues as well.
As of around 12:30 PM EST things are reportedly improving on the Meta/Facebook end of things.
-
@bmeeks Could be in my case. But the OP started his thread 17 hours ago, so that seems unlikely for him. I’ll do some additional testing.
-
@bmeeks Probably unrelated, but I feel compelled to mention since this started for me around the same time (about 5 or 6 PM Eastern time yesterday in the U.S.). I've used unbound with forwarding mode disabled for many years with the outgoing interfaces set my VPN provider client interfaces. I've never had a problem. Yesterday, at the time mentioned, DNS resolution began failing on two independent pfSense machines that I administer that are configured in this way.
I found that if I switched the outgoing interface to WAN (i.e. don't route the queries via the VPN tunnels) it worked again. But since I didn't want this, I enabled forwarding mode and configured system DNS servers, routed via the VPN interfaces, and that's working for me. But I have no idea what changed. From outward appearances, it's as if all the root servers stopped accepting queries from my VPN provider.
-
@TheNarc Is it a more general Internet issue (routing?) that is causing the issue - including meta's challenges?
-
@keyser I wish I knew. I'm not really sure how to diagnose my issue further. Right now I plan to just try switching back to non-forwarding mode in a day or two and see whether it's still broken. I feel like I'd be able to find some evidence of others reporting similar issues if the root servers have just blocked my VPN provider. But I also think whatever is going on in my case must be external to my configuration, both because it did not coincide with any changes I made, and because it began occurring simultaneously on two separate pfSense machines I administer in two separate physical locations.
-
@TheNarc I would tend to agree, but for me coincided with a ISP change (but no pfSense changes - WAN = DHCP).
I'm still unable to poperly resolve names using DOT from pfSense to Quad9. So it's probably something related to TLS sessions/throtteling or whatever.
EDIT: I have no issues using Quad9 as a regular DNS (Not as DOT from pfsense)
-
@TheNarc Hmm, it seems it's related to quad9´s regular DNS name records not resolving correct. Depending on where I resolve, dns.quad9.net does not resolve.
That is needed to resolve for DOT to work (used for certificate verification in TLS setup)Love the no fuss of using the official appliances :-)
-
@keyser Interesting . . . it certainly feels like these things should be related somehow, but I certainly can't definitively tie them together. I'll report back when I try turning off forwarding mode again in a day or two (maybe later tonight if I get impatient, although I tried earlier today and it was still broken).
