DNS suddenly broken [on some VLANs]

johnpoz

@TheNarc said in DNS suddenly broken [on some VLANs]:

left unbound in forwarding mode

You have turned off dnssec right - forwarding and dnssec is combination for failure.

But sure in general concept, if they are intercepting traffic which if you are doing a directed query to 1.2.3.4 and just asking for www.google.co you should get an answer there is redirection going on for sure. Then yeah it should work.. In theory, but we are not sure exactly what they are doing.. You would have to figure out where is the servfail coming from, if you are doing dnssec and you forward and that gets messed up it could be unbound saying yeah hey buddy that stuff isn't passing dnssec, servfail.

For example - this fails dnssec, which I am resolving and using

What all kinds of weirdness can happen when you are being redirected and also asking for dnssec - I have never gone down that rabbit hole to the end to see exactly where it fails, etc.. But its a bad combination..

You could sniff on pfsense and see what is actually being asked, and what gets answered or doesn't get answered, or what might get answered in the dnssec chain that is failing to why your unbound being told to do dnssec when it forwards might say it failed with a servfail.. Which is generic answer and doesn't say exactly what failed, just that something did.

Are you forwarding to a resolver, or just another forwarder? there are quite a few variables at play.. And we don't know what exactly they are doing and how they are doing it.. But doing a directed query to 1.2.3.4 and getting an answer should work unless someone is messing with the query somewhere in the path. Because that IP doesn't answer dns.. Try some other IP you know for a fact doesn't answer dns etc..

Here is an IP for www.netgate.com, its not answer to dns

dig @199.60.103.30 www.google.com
;; communications error to 199.60.103.30#53: timed out
;; communications error to 199.60.103.30#53: timed out
;; communications error to 199.60.103.30#53: timed out

; <<>> DiG 9.18.24-1+ubuntu22.04.1+deb.sury.org+1-Ubuntu <<>> @199.60.103.30 www.google.com
; (1 server found)
;; global options: +cmd
;; no servers could be reached

Do you get a query asking them through your vpn?

TheNarc

@johnpoz I have made sure to disable DNSSEC when I've got unbound in forwarding mode. I'll try to gather some more data over the weekend and report back, and also see whether I can extract any information from Nord's support.

This post is deleted!

TheNarc

@just_a_user_34721 Thanks for providing more corroboration. I've still found no evidence on their site of an admission to this change. It's rather bizarre. I don't have much new data yet either. I'm going to try to test some more if I can think of things that seem valuable to try. One strange thing, though, was that I figured if all my DNS queries are going to be forwarded to Nord's DNS servers anyway, why not just set them as my system DNS servers that unbound will forward to? So I did (103.86.96.100 and 103.86.99.100) and that broke DNS entirely (SERVFAIL on every query).

wfx

Hi all,

I tried a few more tests over the weekend. I set the resolver to forwarding mode. Like other posters DNS started working again.

I initially used NordVPN's DNS servers in general settings but then switched to Quad9. It kept working but they could obviously be redirecting the requests to their own servers.

I also tried a few dig commands suggested by @johnpoz to check for redirection of requests made to root servers. Requests were redirected.

I tried dig @1.2.3.4 netgate.com and it resolved so that was clearly redirected however I then tried to set 1.2.3.4 as sole DNS server in pfSense general settings and DNS stopped working. I would have thought it would also get redirected.

I think that, considering multiple people have had the same problem at the same time, it's quite obvious NordVPN has changed their servers' configuration and are now preventing the use of DNS resolvers in recursive mode (despite their tech support claiming no change was made!!!) and redirecting DNS requests to their own servers.

I am about to write a "please explain" email to them. I'll report back what their answer is.

Thanks to all who posted. I've learned quite a bit as a result of the discussion.

TheNarc

@wfx Thanks for the update. Sounds like you're seeing the same behavior as me. I too expected setting the system DNS to 1.2.3.4 would work since it seems clear that everything is being redirected, but also observed that doing so broke resolution. In my case though, even more surprisingly, setting the system DNS to Nord's DNS servers also broke resolution. I need to set the system DNS servers to valid, non-Nord servers . . . and yet testing suggests that all queries are being redirected. So I'm really not sure what to think at this point.

johnpoz

@TheNarc when you set them in pfsense dns - are they actually using the vpn for the connection. If your not that would explain whey 1.2.3.4 that is being reddirected wouldn't work, and also why you can't talk to the nord dns, because you have to be on their network to talk to them.

TheNarc

@johnpoz As far as I know they should be; I made sure to set their gateways to Nord when I added them. Although I don't even think that setting would have mattered, because I also had my resolution behavior set to use local, ignore remote. So unless I misunderstand that, I think it would mean that all queries would go to unbound, which is configured to only use the Nord interfaces for outgoing, and is also configured to forward to the system DNS servers (which were 1.2.3.4 or the Nord DNS servers in the two scenarios I tested that did not work).

johnpoz

@TheNarc here is the thing should be is nice but verify verify verify is networking matra you should live by

And not using explains what your seeing

TheNarc

@johnpoz Fair enough. So I re-ran some testing, and did verify that the requests were being forwarded to 1.2.3.4 via my Nord interfaces (by examining states). However, they were being forwarded on port 853 because I had unbound's "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" option enabled. As soon as I disabled that, I began resolving successfully again even though I've got 1.2.3.4 set as my only DNS server.

Note that I did not need to disable DNS over TLS when I had valid DNS servers configured in System -> General Setup. But I recalled that when I set Nord's DNS servers there, I was also failing to resolve anything. So next I tried setting my DNS servers back to Nord's, but leaving DNS over TLS for forwarding servers disabled . . . and that worked.

So, would I be off-base in interpreting these test results as:

Nord's DNS servers do not support DNS over TLS
Nord is redirecting DNS, but apparently only when you attempt to use a "bogus" DNS server (e.g. 1.2.3.4). Because otherwise, given my first point, I should fail to resolve anything when I set my DNS server as, say, 8.8.8.8 and enable DNS over TLS for forwarding. But that works.

I guess I could imagine their DNS servers not bothering to support TLS, because they only allow connections to them from clients already connected to their VPN. But it's not clear to me what their motivation would be for this (apparent) selective redirection. Though like I said, maybe I'm way off in my interpretation of what they're doing now . . .

Quick update: running nmap -p 853 on Nord's two DNS servers (103.86.96.100 and 103.86.99.100) showed the port as open, so I attempted to resolve using them with DNS over TLS (+tls arg to dig) going straight out my WAN (verified by checking states) and that worked. So it seems that Nord's servers do support DNS over TLS and do not restrict queries to only clients connected to their VPN. That kind of blows away the theory I put forth for why the Nord DNS servers were not working when I was forwarding to them with DNS over TLS enabled, and I don't yet have a new theory.

DaveP 0

I've the same issue with Nord VPN.

I 'talked' to one of their support staff earlier and they acknowledge the issue and say it is being worked on. I've to keep monitoring and get back if it persists. As this has been ongoing for a week or so I suspect I will be in touch with them again on Monday.

Pity really as up to now I've not had a issue with them. But if it isn't fixed then I'll be moving on. I see AirVPN which I've never heard of is recommended above. Can anyone confirm that I can set that up on PfSense as I have Nord. I don't want to make any assumptions

johnpoz

@DaveP-0 there should be zero reason why you couldn't setup any vpn service with pfsense. If they are using openvpn - your using the openvpn client - why wouldn't you be able to connect?

DaveP 0

@johnpoz Sorry missed your reply as was busy at home. There isn't a problem with OpenVPN it works fine as per the NordVPN setup.

The problem is that I can forward to a DNS provider such as my ISP, QUad 9 or something and it works fine. If I try and bypass these and go to the root servers to build up my own DNS values it fails and DNS will not return any values. Above it says that this is an issue with Nord, which I am using, I've chased them about this but so far have no response on it.

Regards,

TheNarc

@DaveP-0 I think @johnpoz was referring to your question about AirVPN, just saying that so long as AirVPN uses openvpn (which pretty much all VPN providers offer) then you should have no problem setting up a client connection in pfSense.

As to Nord, I have seen more references popping up confirming this issue, but still no real details or information about a possible fix, so I'll just be using forwarding mode until/if it's resolved (pun intended).

johnpoz

@TheNarc yeah there for sure clearly something going on with nord, they are clearly intercepting dns traffic... Smoking gun in another thread when did a directed query to 1.2.3.4 and got an answer.. 1.2.3.4 doesn't do dns that is for damn sure!

DaveP 0

@TheNarc Duh! You are correct I didn't read it correctly. I've turned off the autorenew with Nord and it is up for renewal in a few months. That gives me to to find a new provider if Nord is causing this which it appears. I've written to them now via email and lets see what they say to that email.

DaveP 0

I went reading their blurb about DNS servers and found this sentence

NordVPN offers private DNS in its apps, which ensures tighter security and privacy for your traffic. The NordVPN native applications automatically use NordVPN's DNS servers when connected to VPN. This prevents DNS leaks during your VPN connection, ensuring that your NordVPN private DNS requests are safe.

Maybe what they mean by 'The NordVPN native applications automatically use NordVPN's DNS servers when connected to VPN' is 'We intercept your DNS queries and securely process them via our servers when you are connected via our VPN'. That could be why the automagically send them through normally.

So they are intercepting them and turning off forwarding in PfSense and making it request the root severs can't give an IP as it doesn't know the full domain you have requested just the first character for the root server so it is screwed and returns nothing of use. In which case you would think that their support staff would advise that this is unlikely to get fixed and they see it as a feature. In which case I need to change VPN provider if i want that functionality. OR I reroute DNS queries so they don't go into the VPN tunnel but go directly out via my ISP where accessing the root servers works.

So lets see what Nord say. I've a while before I am due to renew my Nord subscription but I have stopped auto renew in the system.

johnpoz

@DaveP-0 unless you were using qname minimization, and had strict set it should fall back.. If your not doing that the roots do get asked for the fqdn your asking for, but they just respond with the NS for the TLD..

With qname min it would only send them the .tld, but without stict it will fall back to fqdn, etc..

so you would send .com to roots, for example and then domain.com to the qtld servers root handed you, and then only when talking to NS for the domain.tld would you send the fqdn like www.domain.tld

I have this set, but not strict - because that is known to break some major players that have crazy daisy chained names setup, etc.

but more likely that dnssec is just failing if they are intercepting.. You could try turning that off and stil resolving - and try turning off query name min settings if you have those set in unbound advanced settings.

But I don't want any service intercepting anything of mine!! If you want to provide some so call "private" dns for me to use great - I will set my stuff to use those if I want to.. But don't go calling dns interception something your doing to help me!! Cuz you know that is not the reason your doing it!!! Your app, sure ok - but if I just create a connection with my own device via a openvpn settings - you better not be intercepting my dns traffic..

DaveP 0

@johnpoz Sorry I only speak English. No speaky networky things. Just joking but only a little bit.

I'm not a networking person but I have a technical background where networking is the weakest bit. I know enough to set things up but not enough to really debug more than the basic things. I didn't sacrifice enough small furry animals to dark gods when I was learning networking. Plus it was easier back then with coaxial cable and 10-Base-T connections for business use and I had a dial up modem.

I have turned off dnssec because it was recommended to remove something that is known to create issues. query name min settings was turned off in my setup already.

The system worked fine without forward and performing its own lookups with DNSSEC on and query minimum name settings off until a few weeks ago. Of course because I play with things I always assume its something I have done and I'd just installed PfBlockerNG which is a DNS filter so I didn't have to think hard about whose fault it was. I then take everything back to basics to only find it was still going wrong. What a pain that was.

I also don't want people monitoring or intercepting me which is why I set this complex system up in the first place. It is frustrating that this is going on with the provider i use for a security product.

johnpoz

@DaveP-0 said in DNS suddenly broken [on some VLANs]:

provider i use for a security product.

Not sure where you got the idea that vpn services were a security product? They sell you the idea that we won't sell your info to someone else.. Do you really believe that????? really???

I never understood the logic in hey my isp knows where I go, so let me hide that from that that from them and give it to someone else and also pay them money to hide it from the first guy??

But hey if you want to pay company X to connect you to the internet, but then hey they might know where I go, so lets pay company Y to hide that info, because you know company Y is way more trust worthy, and I pay them 3 bucks a month to slow down my internet and and hide where I go from company X.

And then let me complain when this service that is only out for my best interest isn't...
.