DNS suddenly broken [on some VLANs]
-
@johnpoz I will yes. Although I walked away from the computer and won't be back for an hour or two (on a phone now) and also want to run another test. Because as so often happens, while on the treadmill and left to my own thoughts, I realized I'm stupid once again. Someone is absolutely redirecting my DNS, but that person is me. Or at least I'm first in line. I've got a port forward to prevent LAN clients from going around the VPN, and wouldn't you know it, I forgot about it until just now. So Nord may well be redirecting me too, but all my test proved was that I'm an idiot, and I'm redirecting me. As soon as I can ill disable my redirect rule, test again, and post results. Should be able to within two hours.
-
@TheNarc that is actually good info to know.. Since maybe others enabled redirection? So yeah if your redirecting yourself - you would see the same sort of tells of redirection your provider or vpn service was doing it..
-
@Generally-Lost said in DNS suddenly broken [on some VLANs]:
dig @8.8.8.8 netgate.com, which still failed to resolv
Well a directed query like that shouldn't fail - unless you were blocking it yourself, or upstream they were blocking it.. That would have nothing to do with the root servers blocking anything for sure.. Because that query asks 8.8.8.8 hey look this up for me, or hand me whatever you have in your cache for it.. So you get a timeout, a nx a servfail a refused?
-
This post is deleted! -
@johnpoz Okay, just tested with my redirection port forward rules disabled, and the results were the same. So I think it's fairly conclusive that Nord decided to start hijacking all DNS without notifying any of their customers, but at least it seems like we know what's going on.
Edit: One other (maybe?) interesting data point. I was expecting that if Nord is redirecting all DNS that if I set my system DNS server to 1.2.3.4 and left unbound in forwarding mode, DNS resolution would still work. But it doesn't; that configuration yields SERVFAIL. Likely a misconception or misunderstanding on my part, but does that still track with the theory here?
-
OK, got my new box up and running. I switched to AirVPN, and the DNS resolutions are working flawlessly. Shout out to @Uglybrian for the recommendation.
-
@TheNarc said in DNS suddenly broken [on some VLANs]:
left unbound in forwarding mode
You have turned off dnssec right - forwarding and dnssec is combination for failure.
But sure in general concept, if they are intercepting traffic which if you are doing a directed query to 1.2.3.4 and just asking for www.google.co you should get an answer there is redirection going on for sure. Then yeah it should work.. In theory, but we are not sure exactly what they are doing.. You would have to figure out where is the servfail coming from, if you are doing dnssec and you forward and that gets messed up it could be unbound saying yeah hey buddy that stuff isn't passing dnssec, servfail.
For example - this fails dnssec, which I am resolving and using
What all kinds of weirdness can happen when you are being redirected and also asking for dnssec - I have never gone down that rabbit hole to the end to see exactly where it fails, etc.. But its a bad combination..
You could sniff on pfsense and see what is actually being asked, and what gets answered or doesn't get answered, or what might get answered in the dnssec chain that is failing to why your unbound being told to do dnssec when it forwards might say it failed with a servfail.. Which is generic answer and doesn't say exactly what failed, just that something did.
Are you forwarding to a resolver, or just another forwarder? there are quite a few variables at play.. And we don't know what exactly they are doing and how they are doing it.. But doing a directed query to 1.2.3.4 and getting an answer should work unless someone is messing with the query somewhere in the path. Because that IP doesn't answer dns.. Try some other IP you know for a fact doesn't answer dns etc..
Here is an IP for www.netgate.com, its not answer to dns
dig @199.60.103.30 www.google.com ;; communications error to 199.60.103.30#53: timed out ;; communications error to 199.60.103.30#53: timed out ;; communications error to 199.60.103.30#53: timed out ; <<>> DiG 9.18.24-1+ubuntu22.04.1+deb.sury.org+1-Ubuntu <<>> @199.60.103.30 www.google.com ; (1 server found) ;; global options: +cmd ;; no servers could be reachedDo you get a query asking them through your vpn?
-
@johnpoz I have made sure to disable DNSSEC when I've got unbound in forwarding mode. I'll try to gather some more data over the weekend and report back, and also see whether I can extract any information from Nord's support.
-
This post is deleted! -
@just_a_user_34721 Thanks for providing more corroboration. I've still found no evidence on their site of an admission to this change. It's rather bizarre. I don't have much new data yet either. I'm going to try to test some more if I can think of things that seem valuable to try. One strange thing, though, was that I figured if all my DNS queries are going to be forwarded to Nord's DNS servers anyway, why not just set them as my system DNS servers that unbound will forward to? So I did (103.86.96.100 and 103.86.99.100) and that broke DNS entirely (SERVFAIL on every query).
-
Hi all,
I tried a few more tests over the weekend. I set the resolver to forwarding mode. Like other posters DNS started working again.
I initially used NordVPN's DNS servers in general settings but then switched to Quad9. It kept working but they could obviously be redirecting the requests to their own servers.
I also tried a few dig commands suggested by @johnpoz to check for redirection of requests made to root servers. Requests were redirected.
I tried dig @1.2.3.4 netgate.com and it resolved so that was clearly redirected however I then tried to set 1.2.3.4 as sole DNS server in pfSense general settings and DNS stopped working. I would have thought it would also get redirected.
I think that, considering multiple people have had the same problem at the same time, it's quite obvious NordVPN has changed their servers' configuration and are now preventing the use of DNS resolvers in recursive mode (despite their tech support claiming no change was made!!!) and redirecting DNS requests to their own servers.
I am about to write a "please explain" email to them. I'll report back what their answer is.
Thanks to all who posted. I've learned quite a bit as a result of the discussion.
-
@wfx Thanks for the update. Sounds like you're seeing the same behavior as me. I too expected setting the system DNS to 1.2.3.4 would work since it seems clear that everything is being redirected, but also observed that doing so broke resolution. In my case though, even more surprisingly, setting the system DNS to Nord's DNS servers also broke resolution. I need to set the system DNS servers to valid, non-Nord servers . . . and yet testing suggests that all queries are being redirected. So I'm really not sure what to think at this point.
-
@TheNarc when you set them in pfsense dns - are they actually using the vpn for the connection. If your not that would explain whey 1.2.3.4 that is being reddirected wouldn't work, and also why you can't talk to the nord dns, because you have to be on their network to talk to them.
-
@johnpoz As far as I know they should be; I made sure to set their gateways to Nord when I added them. Although I don't even think that setting would have mattered, because I also had my resolution behavior set to use local, ignore remote. So unless I misunderstand that, I think it would mean that all queries would go to unbound, which is configured to only use the Nord interfaces for outgoing, and is also configured to forward to the system DNS servers (which were 1.2.3.4 or the Nord DNS servers in the two scenarios I tested that did not work).
-
@TheNarc here is the thing should be is nice but verify verify verify is networking matra you should live by
And not using explains what your seeing
-
@johnpoz Fair enough. So I re-ran some testing, and did verify that the requests were being forwarded to 1.2.3.4 via my Nord interfaces (by examining states). However, they were being forwarded on port 853 because I had unbound's "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" option enabled. As soon as I disabled that, I began resolving successfully again even though I've got 1.2.3.4 set as my only DNS server.
Note that I did not need to disable DNS over TLS when I had valid DNS servers configured in System -> General Setup. But I recalled that when I set Nord's DNS servers there, I was also failing to resolve anything. So next I tried setting my DNS servers back to Nord's, but leaving DNS over TLS for forwarding servers disabled . . . and that worked.
So, would I be off-base in interpreting these test results as:
- Nord's DNS servers do not support DNS over TLS
- Nord is redirecting DNS, but apparently only when you attempt to use a "bogus" DNS server (e.g. 1.2.3.4). Because otherwise, given my first point, I should fail to resolve anything when I set my DNS server as, say, 8.8.8.8 and enable DNS over TLS for forwarding. But that works.
I guess I could imagine their DNS servers not bothering to support TLS, because they only allow connections to them from clients already connected to their VPN. But it's not clear to me what their motivation would be for this (apparent) selective redirection. Though like I said, maybe I'm way off in my interpretation of what they're doing now . . .
Quick update: running
nmap -p 853on Nord's two DNS servers (103.86.96.100 and 103.86.99.100) showed the port as open, so I attempted to resolve using them with DNS over TLS (+tlsarg to dig) going straight out my WAN (verified by checking states) and that worked. So it seems that Nord's servers do support DNS over TLS and do not restrict queries to only clients connected to their VPN. That kind of blows away the theory I put forth for why the Nord DNS servers were not working when I was forwarding to them with DNS over TLS enabled, and I don't yet have a new theory. -
D DaveP 0 referenced this topic on
-
I've the same issue with Nord VPN.
I 'talked' to one of their support staff earlier and they acknowledge the issue and say it is being worked on. I've to keep monitoring and get back if it persists. As this has been ongoing for a week or so I suspect I will be in touch with them again on Monday.
Pity really as up to now I've not had a issue with them. But if it isn't fixed then I'll be moving on. I see AirVPN which I've never heard of is recommended above. Can anyone confirm that I can set that up on PfSense as I have Nord. I don't want to make any assumptions
-
@DaveP-0 there should be zero reason why you couldn't setup any vpn service with pfsense. If they are using openvpn - your using the openvpn client - why wouldn't you be able to connect?
-
@johnpoz Sorry missed your reply as was busy at home. There isn't a problem with OpenVPN it works fine as per the NordVPN setup.
The problem is that I can forward to a DNS provider such as my ISP, QUad 9 or something and it works fine. If I try and bypass these and go to the root servers to build up my own DNS values it fails and DNS will not return any values. Above it says that this is an issue with Nord, which I am using, I've chased them about this but so far have no response on it.
Regards,
-
@DaveP-0 I think @johnpoz was referring to your question about AirVPN, just saying that so long as AirVPN uses openvpn (which pretty much all VPN providers offer) then you should have no problem setting up a client connection in pfSense.
As to Nord, I have seen more references popping up confirming this issue, but still no real details or information about a possible fix, so I'll just be using forwarding mode until/if it's resolved (pun intended).
