Client DNS doesn't resolve when using VIP in place of interface IP

ttmcmurry

I had this problem on 2.4.5 and now on 2.5. Figured I'd wait until 2.5 was out before I made a post about this.

My inside interface is set to 192.168.1.1 and I created a VIP on .254. When I set a client device to use .254 as the gateway, I can verify the traceroute to the internet and connectivity to the internet is verified.

The problem is when I set the client device to use the VIP as the DNS host. No responses come back. All DNS entries fail to resolve. If I use nslookup and alternate between .1 (inside interface) and .254 (inside VIP) for queries, .1 always works and .254 always fails.

The configuration on the DNS Resolver is set to "All" for "Network Interfaces" - as well as for "Outgoing Network Interfaces". When I look at the list of interfaces in "Network Interfaces" I can clearly see the VIP is listed. I have no ACLs specified in the DNS Resolver's Access Lists.

To test to see if the firewall is processing DNS packets for .254, I created two firewall permit rules on the inside interface to allow port 53 & and 853 against the VIP. The goal is to be able to see each packet as it is processed and permitted. However, pfSense records zero packets, zero data, and zero states.

I'm at a loss for the behavior across two different versions of pfSense.

viktor_g

@ttmcmurry
I can confirm:

dig a mail.ru @192.168.88.44
;; reply from unexpected source: 192.168.88.41#53, expected 192.168.88.44#53

192.168.88.41 - LAN IP, 192.168.88.44 - LAN CARP VIP

It's better to port forward DNS queries in this case,
see https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html

viktor_g

Redmine issue: https://redmine.pfsense.org/issues/11457

ttmcmurry

@viktor_g
Ever have one of those moments when you're about to fall asleep, then your brain goes "I know what the problem is and how to fix it"?

I realized I was making an assumption that DNS Resolver's Network Interfaces "All" option really includes ALL interfaces listed in the box.

Then I shift-selected all of the listed interfaces, excluding all, clicked save and then ..

> server 192.168.1.1
Default Server:  pfSense.tnthome.local
Address:  192.168.1.1

> www.honda.com
Server:  pfSense.tnthome.local
Address:  192.168.1.1

Non-authoritative answer:
Name:    e8624.x.akamaiedge.net
Address:  23.1.124.189
Aliases:  www.honda.com
          www.honda.com.edgekey.net

> server 192.168.1.254
Default Server:  [192.168.1.254]
Address:  192.168.1.254

> www.bentleymotors.com
Server:  [192.168.1.254]
Address:  192.168.1.254

Non-authoritative answer:
Name:    e1838.a.akamaiedge.net
Address:  23.46.250.55
Aliases:  www.bentleymotors.com
          san.bentleymotors.com.edgekey.net

This looks like a logic bug in either the UI or code path to Unbound. The UI is certainly showing all of the interfaces I expect, but "All" isn't including everything displayed in the list.

I checked to see the documentation if VIPs were intentionally excluded. If that is the case, it is not documented. My assumption is (1) the documentation is accurate and (2) DNS should work on "All" interfaces, including VIPs.

I'll update the bug in Redmine.

jimp

I can't reproduce this here. Queries to VIPs are returned properly. There must be something else in your config contributing to it.

ttmcmurry

@jimp
The setting causing the problem with "All" is Enabling SSL/TLS Service on DNS Resolver.

SSL: Off / NI: All - VIP Works
SSL: Off / NI: Manual - VIP Works
SSL: On / NI: Manual - VIP Works
SSL: On / NI: All - VIP Fails

jimp

OK, now with that set I can reproduce it.

Looks like the reason is due to our setting interfaces-automatic: no when SSL is on, which used to be necessary.

A quick check now seems that it still is OK with it set to yes with SSL, but I need to test more.

jimp

Ref: https://redmine.pfsense.org/issues/8030#note-6

ttmcmurry

@jimp

Okay, and thank you. After reading the official documentation, it doesn't discuss these matters - since my actions are based upon it, I wanted to share those findings.

I'm noticing is the DNS over TLS feature is not mentioned in the DNS Resolver or Advanced DNS Resolver doc. If there are any limits placed upon using SSL/TLS, it is not covered there or linked to the actual DNS over TLS document. The reader must perform a search through documentation to find that page. Luckily, the DNS over TLS doc links back to DNS Resolver, but the opposite is not true.

When scanning through the DNS over TLS doc, nothing is mentioned about limitations on VIPs or the All network interface in the DNS Resolver.

I'm trying to be studious and make sure I'm reading all of this correctly.

jimp

Try this patch and see if both regular and TLS DNS queries work:

diff --git a/src/etc/inc/unbound.inc b/src/etc/inc/unbound.inc
index 531b906c74..28f41cd188 100644
--- a/src/etc/inc/unbound.inc
+++ b/src/etc/inc/unbound.inc
@@ -228,7 +228,7 @@ EOF;
        if (empty($unboundcfg['active_interface']) || in_array("all", $active_interfaces, true)) {
                $bindints[] = "0.0.0.0";
                $bindints[] = "::0";
-               $bindintcfg .= "interface-automatic: " . (isset($unboundcfg['enablessl']) ? "no" : "yes") . "\n";
+               $bindintcfg .= "interface-automatic: yes\n";
        } else {
                foreach ($active_interfaces as $ubif) {
                        /* Do not bind to disabled/nocarrier interfaces,

ttmcmurry

@jimp

Applied patch, restarted unbound. Tested the same 4 scenarios. They all work now with the VIP with SSL/TLS over DNS enabled.

SSL: Off / NI: All - VIP Works
SSL: Off / NI: Manual - VIP Works
SSL: On / NI: Manual - VIP Works
SSL: On / NI: All - VIP Works

jimp

Did you try TLS-enabled queries or just regular ones? Or both?

jimp

Unfortunately that appears to break DNS over TLS service. With it set to yes, unbound does not bind to port 853.

So for now we may just have to chalk this up to a limitation of unbound. Might need to add a warning in the GUI and docs at least.

ttmcmurry

@jimp said in Client DNS doesn't resolve when using VIP in place of interface IP:

Did you try TLS-enabled queries or just regular ones? Or both?

I tested regular over 53. I'm switching to ubuntu to test dns over tls via DIG or kdig/knot.

jimp

I tested with kdig but that wasn't necessary, really. Checking sockstat | grep unbound on the firewall showed it wasn't bound to 853. When I reverted the change and let it stay on no, it bound to 853 and a kdig query worked.

There is already a warning in the GUI about this under the Enable SSL/TLS Service option but perhaps it could be stronger or better worded.

The DNS Resolver docs haven't been updated yet with any of these settings, but when I get to that point I can add stronger warnings as well.

ttmcmurry

@jimp

Confirmed your results 100% the same with 53/853 with ssl yes/no via the patch.

When I revert the patch, then manually specify the DNS Resolver Network Interfaces with SSL/TLS enabled, both 53 & 853 requests resolve on a VIP; 853 binds to unbound.

Thank you for your time on this, I understand what's happening and how to address it. At least I took time to RTFM before posting in the forum .

jimp

If nothing else it was worth checking again if unbound solved that limitation in the last 3 years

johnpoz

@ttmcmurry said in Client DNS doesn't resolve when using VIP in place of interface IP:

853 binds to unbound.

While this is a off topic question.. I am curious.. In what scenario would you be using dot locally? I don't get the use case..

Why would you want client or even downstream dns in your own network to use dot? Who on your own network would be intercepting or interfering with your dns. I really for the life of me see no valid use case for doing this on a local network.. Is it to remove some stupid warning on a device that your local network is not secure - because its not using dot or doh?

I could see the functionality for unbound, if it was used as an upstream dns where the clients of said dns transit to it over over a public or hostile network. But I don't get it why this is something you would want to do over you own local network to your own local dns.

I guess it should work - if so desired ;) But I don't see use case, other than lab or play..

ttmcmurry

@johnpoz

For DoT - In my case, it's pure lab work. Networks behind networks, behind application delivery appliances & virtual routers. I'm also testing Win10's native DoT capability for plans down the road - not just browser-based DoT that's present in Chrome and Edge, which works a bit differently than at the OS level.

I completely understand why using DoT inside a private network may be viewed as unnecessary. Generally speaking, I see more apps in the wild attempting to use DoT opportunistically - particularly in mobile device apps. The thought here is have it working anyway in case a device starts using it.

For upstream DoT - and I'm calling out AT&T right now - their Gateways intercept and rewrite DNS queries to go through their DNS servers - purely for the purpose of data gathering on customers. The gateways also prohibit changing DNS servers, The only way around it is to use DoT from pfSense or PiHole .. something that sits between the internal network and the gateway itself.

I feel I must mention I work in the financial sector, so it pays to be as forward thinking as possible with security and follow zero trust models in every scenario. At this point, almost all the work I do with networking assumes nothing is trustworthy and never will be - so it is necessary to integrate technologies that look for bad actors at all possible network node locations; utilize SIEM and analysis tools; layer2-7 inspection, etc.

jimp

Usually the cases are:

Situations where you may not trust the security of the local network (e.g. guest network wireless which is unencrypted, or cases where a network is forced to use wireless protocols which may not be fully secure against decrypting traffic from other associated clients)
You want to intercept local client DNS over TLS queries (and hope they don't validate the hostname...)
General paranoia and/or "cool factor" of it

I'm sure there are more, but it isn't something most people will probably need to have enabled.