DNS Redirect Failure

jamestford

Wondering if any one out there has come across this issue. We have a situation where we want to ensure that all computers within a given space are forced to use the pfsense DNS Resolver instead of going directly to google or others for resolution. To accomplish this we have followed the guidance provided in the Netgate article here: https://docs.netgate.com/pfsense/en/latest/dns/redirecting-all-dns-requests-to-pfsense.html

Here are our settings:
1 new NAT Port Forward Rule.
Interface: LAB
Protocol: TCP/UDP
Destination: Invert Match checked, LAB Address
Destination Port Range: 53 (DNS)
Redirect Target IP: 127.0.0.1
Redirect Target Port: 53 (DNS)
Description: Disable External DNS Queries
NAT Reflection: Disable

Upon enabling this port forward we get the following from hosts internal to the LAB which are set to use google dns (8.8.8.8 and 8.8.4.4)

C:\Users\james>nslookup
DNS request timed out.
timeout was 2 seconds.
Default Server: UnKnown
Address: 8.8.8.8

www.google.com
Server: UnKnown
Address: 8.8.8.8

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out

If we disable the port forward rule and run it again we have successful name resolution, any thoughts on what we have misconfigured?

johnpoz

And are you running unbound on pfsense? Do you have it set to listen on loopback?

jamestford

Yes, we are currently running unbound and it is set to ALL for interfaces. A quick netstat shows that it is bound to *.53 for tcp4 and tcp6.

We ran two traces on the LAB interface to see what the difference in traffic was, by the looks of the dump the router is responding but the client still times out. See trace excerpts below.

Trace of normal nslookup with no redirection:
21:42:15.871827 IP 10.10.12.10.55170 > google-public-dns-a.google.com.domain: 8822+ A? www.google.com. (32)
21:42:15.885876 IP google-public-dns-a.google.com.domain > 10.10.12.10.55170: 8822 1/0/0 A 172.217.15.100 (48)
21:42:47.404402 IP 10.10.12.10.62397 > google-public-dns-a.google.com.domain: 35829+ A? array508-prod.do.dsp.mp.microsoft.com. (55)
21:42:47.497374 IP google-public-dns-a.google.com.domain > 10.10.12.10.62397: 35829 2/0/0 CNAME array508-prod.dodsp.mp.microsoft.com.nsatc.net., A 40.79.65.78 (131)

Trace with redirection enabled, you can see the router (10.10.12.1) responding
21:43:55.044378 IP 10.10.12.10.62398 > google-public-dns-a.google.com.domain: 1+ PTR? 8.8.8.8.in-addr.arpa. (38)
21:43:55.044517 IP 10.10.12.1.domain > 10.10.12.10.62398: 1 1/0/0 PTR google-public-dns-a.google.com. (82)
21:44:08.044150 IP 10.10.12.10.62399 > google-public-dns-a.google.com.domain: 2+ A? www.google.com. (32)
21:44:08.044332 IP 10.10.12.1.domain > 10.10.12.10.62399: 2 1/0/0 A 172.217.164.164 (48)
21:44:10.060066 IP 10.10.12.10.62400 > google-public-dns-a.google.com.domain: 3+ AAAA? www.google.com. (32)
21:44:10.060245 IP 10.10.12.1.domain > 10.10.12.10.62400: 3 1/0/0 AAAA 2607:f8b0:4004:814::2004 (60)
21:44:12.081531 IP 10.10.12.10.62401 > google-public-dns-a.google.com.domain: 4+ A? www.google.com. (32)
21:44:12.081751 IP 10.10.12.1.domain > 10.10.12.10.62401: 4 1/0/0 A 172.217.164.164 (48)
21:44:14.103333 IP 10.10.12.10.62402 > google-public-dns-a.google.com.domain: 5+ AAAA? www.google.com. (32)
21:44:14.103555 IP 10.10.12.1.domain > 10.10.12.10.62402: 5 1/0/0 AAAA 2607:f8b0:4004:814::2004 (60)

jamestford

We have isolated the issue but still don't understand why it is behaving the way it is. After hunting around in our test lab we noticed the behavior starts when the option "Respond to incoming SSL/TLS queries from local clients" is enabled. We assumed this allows clients to send requests to either port 53 or 853. We have tested enabling and disabling the option and it is a repeatable failure. Thoughts?

jamestford

DNS over TLS is probably way too much overkill anyway for something that is supposed to be fast and internal using resolver so we have just disabled this feature and things are working as intended.

4o4rh

@jamestford hey James, would you mind to share the final config or what you did to get it working properly please. I have the LANs listening on 53 and blocked 53 on the WAN. All requests go via 853 on the WAN side, this works. and all normal clients work. However, i have a few Android devices e.g. Galaxy S8 which are not respecting the DHCP DNS entry of the pfsense.

These are trying to connect directly to google DNS which of course is blocked. (at least i know that the blocking works)

johnpoz

@gwaitsi said in DNS Redirect Failure:

53 and blocked 53 on the WAN.

Huh?

Follow the instructions linked too to redirect dns.

21:43:55.044517 IP 10.10.12.1.domain > 10.10.12.10.62398:

@jamestford
Not sure what you did exactly - but listening on 853 for tls would have nothing to do with your problem

If you had actually redirected dns per the instructions then you wouldn't see pfsense IP in the sniff you would see the IP you redirected..

02:31:50.630648 IP 192.168.9.100.56750 > 8.8.8.8.53: UDP, length 55
02:31:50.630812 IP 8.8.8.8.53 > 192.168.9.100.56750: UDP, length 59

So a different test will query 172.16.42.42, which is not some dns that could ever answer - but would be redirected to unbound and look like the answer came from that IP..

02:35:16.886717 IP 192.168.9.100.53428 > 172.16.42.42.53: UDP, length 55
02:35:16.886864 IP 172.16.42.42.53 > 192.168.9.100.53428: UDP, length 59

But yeah tls locally is pretty pointless anyway. And sure is not enabled out of the box - so you must of selected that.

jamestford

@johnpoz said in DNS Redirect Failure:

172.16.42.42,

Thanks for the response, wanted to share the traffic we observed when the "Respond to incoming SSL/TLS queries from local clients" feature is disabled versus enabled. I am still baffled as to why this setting would have anything to do with redirection, I'm sure it's something obvious that I've overlooked.

Scenario 1: Lookup of non-existent domain with SSL/TLS disabled. Notice the router does not respond, redirection is transparent and working in the background properly.

14:15:44.507675 IP 10.10.12.10.57610 > google-public-dns-a.google.com.domain: 2+ PTR? 42.42.16.172.in-addr.arpa. (43)
14:15:44.507822 IP google-public-dns-a.google.com.domain > 10.10.12.10.57610: 2 NXDomain* 0/1/0 (102)

Scenario 2: Respond to incoming SSL/TLS queries from local clients is enabled. Redirection is still in place, the only difference is the enablement of responding to SSL/TLS queries. It should have nothing to do with this test but for some weird reason it totally changes the results, see below. The router interjects with responses and the request times out on the client.

14:18:58.518821 IP 10.10.12.10.65143 > google-public-dns-a.google.com.domain: 8+ PTR? 42.42.16.172.in-addr.arpa. (43)
14:18:58.518989 IP 10.10.12.1.domain > 10.10.12.10.65143: 8 NXDomain* 0/1/0 (102)

Here is our current DNS configuration:
Enable DNS Resolver: Yes
Listen Port: 53
Enable SSL/TLS Service: Disabled (Currently, we would like to enable but having issues)
SSL/TLS Certificate: Firewall (ACME Encrypt v2 Cert, resolves correctly in chrome/IE)
SSL/TLS Listen Port: 853
Network Interfaces: ALL
Outgoing Network Interfaces: ALL
System Domain Local Zone Type: Transparent
DNSSEC: Enabled
DNS Query Forwarding: Forwarding Mode Not Enabled, Use SSL/TLS for outgoing DNS Not Enabled
DHCP Registration Enabled
Static DHCP Mappings Enabled
OpenVPN Clients Registration Enabled
Custom Options:
server:
private-domain: "redacted"

Advanced Settings:
Hide Identity Enabled
Hide Version Enabled
Query Name Minimization Enabled
Harden DNSSEC Data Enabled

4o4rh

thanks fellas, for me i had to do the following

DNS Resolver
Enable DNSSEC Support = checked
DNS Query Forwarding = checked
Use SSL/TLS for outgoing DNS Queries to Forwarding Servers = checked

NAT
LAN TCP/UDP * * !LOCAL DNS LAN_INT DNS

LAN
Allow TCP/UDP LOCAL * LOCAL DNS
Allow TPC/UDP LOCAL * !LOCAL 853
Block IP4/6 * * * *

I found using a packet trace, with DNS Query unchecked,
DNS Resolver showed a list of dns entries in the cache, and android clients were successfully connecting to 8.8.8.8:53
With DNS Query checked, the DNS Resolver table shows as 9.9.9.10@853 and a packet trace shows only 853 on the WAN

So for me, all is good now.

johnpoz

@gwaitsi said in DNS Redirect Failure:

Enable DNSSEC Support = checked

If your going to FORWARD that dnssec being enabled its utterly POINTLESS and just adds queries that mean nothing.

"Query Name Minimization Enabled"

Why do you have that enabled - that can cause you problems depending on what your looking for.. I will attempt to duplicate listening on SSL because it has nothing to do with a normal query to 53 being redirected.

4o4rh

@johnpoz think you mix me with james.
i don't have query name minimization enabled.
if i disable "DNS Query Forwarding", although i see DNS entries cached under DNS Resolver, i see 53 traffic going over the WAN interface.
if i enable "DNS Query Forwarding", i don't see any 53 traffic over the wan, but lots of 853

Grimson

@gwaitsi said in DNS Redirect Failure:

if i disable "DNS Query Forwarding", although i see DNS entries cached under DNS Resolver, i see 53 traffic going over the WAN interface.

Yeah, of course. Without forwarding the resolver is doing it's job by asking the appropriate DNS servers for their information. That's how a resolver works:
https://forum.netgate.com/topic/117972/difference-between-dns-resolver-and-dns-forwarder/12

johnpoz

No I was I was talking to the OP.. for that..

My guess to why have issue with redirection and use of tls listener is prob
"Activating this option disables automatic interface response routing behavior, thus it works best with specific interface bindings."

Again - there really is ZERO reason to enable tls on your local secure network anyway... Do you feel there is someone sniffing your dns queries to pfsense and altering them or knowing where your going -- Really??

jamestford

@johnpoz

Totally agree with you on the applicability of this for internal use, the real reason we wanted to enable this is for testing purposes, this is for an internal security test lab used for students to play around with settings and scenarios so they can see the difference between traffic captures. We may try to enable again and specify the lab interface to see if that makes a difference. But I'm pretty happy that the resolver with redirection and DNSSEC is working as intended so that's the main issue successfully tackled.

4o4rh

@johnpoz i think we are talking cross wires man.

local LAN allows 53 queries and should trap DHCP bypasses e.g. Galaxy S8 seems hard coded with 8.8.8.8 (i want pfsense to be sole source of DNS locally 53 is ok)
DNS should be blocked from the WAN
pfsense should connect to DNSSEC servers for queries (if not cached locally).
DNSSEC requests should be directed over the VPN

all those conditions are met, so long as the DNS queries are forwarded to the DNSSEC servers in the general tab. Ideally, i would like to have what you are suggesting, but the DNS Resolver should have a TLS connection to a DNSSEC server.

Grimson

@gwaitsi said in DNS Redirect Failure:

all those conditions are met, so long as the DNS queries for forwarded to the DNSSEC servers in the general tab. Ideally, i would like to have what you are suggesting, but the DNS Resolver should have a TLS connection to a DNSSEC server.

DNS over TLS is not DNSSEC, those are completely different things. DNSSEC is only really useful if you are resolving instead of forwarding. Inform yourself.

4o4rh

@grimson I am connecting to DNSSEC servers Quad9 and Cloudfare with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and therefore doing both.

Only, without forwarding the connection, dns requests are simply pass through the WAN. Didn't see your earlier post with the link.Forwarding is in my case, is therefore the desired option.

Grimson

@gwaitsi said in DNS Redirect Failure:

I am connecting to DNSSEC servers Quad9 and Cloudfare with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and therefore doing both. Only, without forwarding the connection, dns requests are simply pass through the WAN.

Then remove WAN from the "Outgoing Network Interfaces" section of the resolver settings if you don't want DNS to go through your WAN. DNSSEC with forwarding is pointless, as the servers you forward too can manipulate all the data and you have to utterly trust them.

4o4rh

@grimson problem there is, i need DNS on the WAN as well.

I need DNS on the WAN to establish VPNs both from pfsense and from
clients that have their own VPNs and go out via the WAN
China and this forum for example don't allow connections from my VPN provider

Grimson

@gwaitsi said in DNS Redirect Failure:

@grimson problem there is, i need DNS on the WAN as well.

I need DNS on the WAN to establish VPNs both from pfsense and from
clients that have their own VPNs and go out via the WAN

You only need the initial connection to at least one VPN from pfSense, from there on the resolver can do it's job no matter how the rest of the traffic is then routed. For this initial connection you can use IPs instead of domain names.

If your VPN provider forces you to use domain names you can go to the general settings, tick "Disable DNS Forwarder" and add one or two DNS servers. Then pfSense will use these DNS servers by itself while clients can still be forced to use the resolver, you also might have to manually specify the resolver as DNS in the DHCP server settings in that case.