Possible bug in DNS Resolver.

mrzaz

Hello,

During investigation of another problem I think I may have stumbled on a possible bug related to DNS Resolver and Forwarding mode.
After reviewing the DNS Hangout with Jim P I thought I had it covered but saw a strange behaviour during the testing.

According to how I have understood the whole is:

DNS Resolver in Non-Forwarding mode will do a lookup to available root servers on all available WAN.
DNS Resolver in Forwarding mode will do lookup to the forwarding DNS IP defined for each WAN gateway.
eg.
The following is defined in General - DNS Server Settings:
DNS Server 1 8.8.8.8 WANGW - wan - 87.x.x.1
DNS Server 2 8.8.4.4 WAN2MOBILEGW - opt3 - 192.168.125.1

This will send a DNS request to 8.8.8.8 out the WANGW and a DNS request to 8.8.4.4 out on WAN2MOBILEGW.
Of course, the "Outgoing Network Interfaces" in DNS Resolver must have the NIC for WANGW and WAN2MOBILEGW selected. (WAN + WAN2MOBILE in my case)

What I have seen is that even if I define one DNS to only one Gateway, the request is sent to both DNS IPs defined
on all interfaces that has been defined in DNS Resolver as outgoing.

eg. 8.8.8.8 and 8.8.4.4 is sent to both WAN and WAN2
instead of
8.8.8.8 -> WAN
8.8.4.4 -> WAN2
even with this config, it will send out DNS requests to both 8.8.8.8 and 8.8.4.4

This is pcap traces taken on the WAN2MOBILE
23:38:32.516036 IP 192.168.125.2.61987 > 8.8.8.8.53: UDP, length 63
23:38:32.532043 IP 192.168.125.2.51270 > 8.8.4.4.53: UDP, length 53
23:38:32.617246 IP 8.8.8.8.53 > 192.168.125.2.5918: UDP, length 111
23:38:32.679359 IP 192.168.125.2.11608 > 8.8.4.4.53: UDP, length 56
23:38:32.679987 IP 8.8.4.4.53 > 192.168.125.2.27262: UDP, length 149
23:38:32.691486 IP 8.8.4.4.53 > 192.168.125.2.51270: UDP, length 149
23:38:32.691613 IP 8.8.8.8.53 > 192.168.125.2.61987: UDP, length 111
23:38:32.714615 IP 8.8.4.4.53 > 192.168.125.2.11608: UDP, length 220

What I could see in the /etc/resolv.conf is:
nameserver 127.0.0.1
search mrzaz.com
nameserver 8.8.8.8
nameserver 8.8.4.4

I did a small test where I removed all DNS entries in General and then all DNS lookup stopped working from clients.
$ cat /etc/resolv.conf
nameserver 127.0.0.1
search mrzaz.com

Then I added IPs again without selecting any gateways and saw this.
$ cat /etc/resolv.conf
nameserver 127.0.0.1
search mrzaz.com
nameserver 8.8.8.8
nameserver 8.8.4.4

- How is the DNS defined in General actually tied to a specific gateway in FreeBSD (read Ubound) when defined in pfSense?

Btw. I am using Gateway Group and gateway rules for this in "Rules".

I get the feeling that there is a bug lurking but can't be sure.

UPDATED:
I checkeded the routing table and i have the following:
8.8.4.4 192.168.125.1 UGHS 57459 1500 ue1
8.8.8.8 87.96.165.1 UGHS 954168 1500 re0

So there should be static routes forcing the DNS out the correct interface but still I see both 8.8.8.8 and 8.8.4.4 on ue1 in the example.

After some more tests I had to insert a rule that captured the IP 8.8.8.8 / 8.8.4.4 and forced it through the "default" gateway and not DualGW Gateway group.
So after all, it may not be a bug BUT think this should be handled somehow as this could be a trap that is easy to fall in and difficult to spot.

Best regards
Dan Lundqvist
Stockholm, Sweden

jimp

Unlikely to be a bug.

Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)

mrzaz

See the UPDATE section above. Is kind of a trap that is easy to fall in when doing Gateway Groups and DNS Resolver in Forwarding mode.
Maybe some updates or possible autocreated rule to handle this ? Not sure where to place this so the sysadmin is aware and don't go into the pitfall.

//Danne

@jimp:

Unlikely to be a bug.

Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)

mrzaz

Small comment to previous jimps post/reply:

I have static IP on both WAN and WAN2.
DNS Server Override is NOT checked.
DNS Forwarder (dnsmasque) is completely disabled, only DNS Resolver in Forwarding mode is enabled.

After some modification of rules to force GoogleDNS through default routing instead of GatewayGroup I cleared up some problems.

HOWEVER, I still think there is something fishy.

In DNS Resolver I have NOT selected WAN2MOBILE as outgoing interface but I still see traces of outgoing DNS requests to 8.8.4.4
from/on the WAN2MOBILE interface when doing packet capture.

Please see the attached screenshots. Especially the DNS Resolver one. According to this, WAN2MOBILE should NOT be used
for any outgoing DNS from DNS Resolver but still I see outgoing DNS traffic. :-/

14 232.701197 192.168.125.2 8.8.4.4 DNS 86 Standard query 0x2d8f PTR 27.120.68.138.in-addr.arpa
15 233.027563 8.8.4.4 192.168.125.2 DNS 86 Standard query response 0x2d8f Server failure PTR 27.120.68.138.in-addr.arpa
16 233.040433 192.168.125.2 8.8.4.4 DNS 86 Standard query 0x2d8f PTR 27.120.68.138.in-addr.arpa
17 233.114054 8.8.4.4 192.168.125.2 DNS 86 Standard query response 0x2d8f Server failure PTR 27.120.68.138.in-addr.arpa
18 233.341240 192.168.125.2 8.8.4.4 DNS 86 Standard query 0xa56d PTR 39.43.244.104.in-addr.arpa
19 233.432171 8.8.4.4 192.168.125.2 DNS 86 Standard query response 0xa56d Server failure PTR 39.43.244.104.in-addr.arpa
20 233.443919 192.168.125.2 8.8.4.4 DNS 86 Standard query 0xa56d PTR 39.43.244.104.in-addr.arpa
21 233.982914 8.8.4.4 192.168.125.2 DNS 86 Standard query response 0xa56d Server failure PTR 39.43.244.104.in-addr.arpa

I did the trace with and without promiscuous mode enabled with same result.

COMMAND: unbound-control -c /var/unbound/unbound.conf lookup .

The following name servers are used for lookup of .
forwarding request:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 2 IP addresses.
8.8.8.8         	rto 100 msec, ttl 65, ping 24 var 19 rtt 100, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
8.8.4.4         	rto 407 msec, ttl 425, ping 87 var 80 rtt 407, tA 0, tAAAA 0, tother 0, EDNS 0 probed.

COMMAND: cat /etc/resolv.conf

nameserver 127.0.0.1
search xxxxx.com				; masked my real domain with xxxxx
nameserver 8.8.8.8
nameserver 8.8.4.4

COMMAND: cat /var/unbound/unbound.conf (only include a small portion)

# Interface IP(s) to bind to
interface: 192.168.120.20
interface: 2001:470:28:xx::1		; masked my real IP with xx
interface: 127.0.0.1
interface: ::1

# Outgoing interfaces to be used
outgoing-interface: xx.xx.165.51	; masked my real IP with xx
outgoing-interface: 2001:470:27:xx::2	; masked my real IP with xx
.
.
# Forwarding
forward-zone:
	name: "."
	forward-addr: 8.8.8.8
	forward-addr: 8.8.4.4

Did some tracing and saw the following: (more detailed logs could be requested if needed)

Oct 14 00:37:08 pfSense unbound: [78182:2] info: processQueryTargets: 2.android.pool.ntp.org. A IN
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: processQueryTargets: targetqueries 0, currentqueries 0 sentcount 0
Oct 14 00:37:08 pfSense unbound: [78182:2] info: DelegationPoint<.>: 0 names (0 missing), 2 addrs (0 result, 2 avail) parentNS
Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    ip4 8.8.4.4 port 53 (len 16)
Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    ip4 8.8.8.8 port 53 (len 16)
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: attempt to get extra 3 targets
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: servselect ip4 8.8.8.8 port 53 (len 16)
Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    rtt=155
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: servselect ip4 8.8.4.4 port 53 (len 16)
Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    rtt=211
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: selrtt 155
Oct 14 00:37:08 pfSense unbound: [78182:2] info: sending query: 2.android.pool.ntp.org. A IN
Oct 14 00:37:08 pfSense unbound: [78182:2] debug: sending to target: <.> 8.8.4.4#53

DNSResolv1.jpg_thumb

GeneralDNS.jpg_thumb

Routes.jpg_thumb

mrzaz

@jimp:

Unlikely to be a bug.

Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)

As mentioned in my previous post I do not think the routing table is bad and I do not allow any DHCP overide. (not using DHCP. Static on both WANs)
Please see the attached routingtable. Routing_ipv4.jpg

I have masked out some part of the IPs to protect some but you could see the content structure anyway.
120.0 is the LAN net (where .20 = pfSense)
121.0 is OpenVPN
125.0 is a local lan between PFS and a HiLink 4G modem. (that is USBSWITCHED) WAN2MOBILE
x.x.165.0 = WAN1
x.x.50.x remote end of a IPSec

Routing_IPv4.jpg_thumb