Possible bug in DNS Resolver.



  • Hello,

    During investigation of another problem I think I may have stumbled on a possible bug related to DNS Resolver and Forwarding mode.
    After reviewing the DNS Hangout with Jim P I thought I had it covered but saw a strange behaviour during the testing.

    According to how I have understood the whole is:

    • DNS Resolver in Non-Forwarding mode will do a lookup to available root servers on all available WAN.
    • DNS Resolver in Forwarding mode will do lookup to the forwarding DNS IP defined for each WAN gateway.
      eg.
      The following is defined in General - DNS Server Settings:
      DNS Server 1    8.8.8.8    WANGW              - wan - 87.x.x.1
      DNS Server 2    8.8.4.4    WAN2MOBILEGW - opt3 - 192.168.125.1

    This will send a DNS request to 8.8.8.8 out the WANGW and a DNS request to 8.8.4.4 out on WAN2MOBILEGW.
    Of course, the "Outgoing Network Interfaces" in DNS Resolver must have the NIC for WANGW and WAN2MOBILEGW selected. (WAN + WAN2MOBILE in my case)

    What I have seen is that even if I define one DNS to only one Gateway, the request is sent to both DNS IPs defined
    on all interfaces that has been defined in DNS Resolver as outgoing.

    eg. 8.8.8.8 and 8.8.4.4 is sent to both WAN and WAN2
    instead of
    8.8.8.8 -> WAN
    8.8.4.4 -> WAN2
    even with this config, it will send out DNS requests to both 8.8.8.8 and 8.8.4.4

    This is pcap traces taken on the WAN2MOBILE
    23:38:32.516036 IP 192.168.125.2.61987 > 8.8.8.8.53: UDP, length 63
    23:38:32.532043 IP 192.168.125.2.51270 > 8.8.4.4.53: UDP, length 53
    23:38:32.617246 IP 8.8.8.8.53 > 192.168.125.2.5918: UDP, length 111
    23:38:32.679359 IP 192.168.125.2.11608 > 8.8.4.4.53: UDP, length 56
    23:38:32.679987 IP 8.8.4.4.53 > 192.168.125.2.27262: UDP, length 149
    23:38:32.691486 IP 8.8.4.4.53 > 192.168.125.2.51270: UDP, length 149
    23:38:32.691613 IP 8.8.8.8.53 > 192.168.125.2.61987: UDP, length 111
    23:38:32.714615 IP 8.8.4.4.53 > 192.168.125.2.11608: UDP, length 220

    What I could see in the /etc/resolv.conf is:
    nameserver 127.0.0.1
    search mrzaz.com
    nameserver 8.8.8.8
    nameserver 8.8.4.4

    I did a small test where I removed all DNS entries in General and then all DNS lookup stopped working from clients.
    $ cat /etc/resolv.conf
    nameserver 127.0.0.1
    search mrzaz.com

    Then I added IPs again without selecting any gateways and saw this.
    $ cat /etc/resolv.conf
    nameserver 127.0.0.1
    search mrzaz.com
    nameserver 8.8.8.8
    nameserver 8.8.4.4

    - How is the DNS defined in General actually tied to a specific gateway in FreeBSD (read Ubound) when defined in pfSense?

    Btw. I am using Gateway Group and gateway rules for this in "Rules".

    I get the feeling that there is a bug lurking but can't be sure.

    UPDATED:
    I checkeded the routing table and i have the following:
    8.8.4.4 192.168.125.1 UGHS 57459 1500 ue1
    8.8.8.8 87.96.165.1 UGHS 954168 1500 re0

    So there should be static routes forcing the DNS out the correct interface but still I see both 8.8.8.8 and 8.8.4.4 on ue1 in the example.

    After some more tests I had to insert a rule that captured the IP 8.8.8.8 / 8.8.4.4 and forced it through the "default" gateway and not DualGW Gateway group.
    So after all, it may not be a bug BUT think this should be handled somehow as this could be a trap that is easy to fall in and difficult to spot.

    Best regards
    Dan Lundqvist
    Stockholm, Sweden


  • Rebel Alliance Developer Netgate

    Unlikely to be a bug.

    Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

    If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)



  • See the UPDATE section above.  Is kind of a trap that is easy to fall in when doing Gateway Groups and DNS Resolver in Forwarding mode.
    Maybe some updates or possible autocreated rule to handle this ?  Not sure where to place this so the sysadmin is aware and don't go into the pitfall.

    //Danne

    @jimp:

    Unlikely to be a bug.

    Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

    If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)



  • Small comment to previous jimps post/reply:

    • I have static IP on both WAN and WAN2.
    • DNS Server Override is NOT checked.
    • DNS Forwarder (dnsmasque) is completely disabled, only DNS Resolver in Forwarding mode is enabled.

    After some modification of rules to force GoogleDNS through default routing instead of GatewayGroup I cleared up some problems.

    HOWEVER, I still think there is something fishy.

    In DNS Resolver I have NOT selected WAN2MOBILE as outgoing interface but I still see traces of outgoing DNS requests to 8.8.4.4
    from/on the WAN2MOBILE interface when doing packet capture.

    Please see the attached screenshots. Especially the DNS Resolver one. According to this, WAN2MOBILE should NOT be used
    for any outgoing DNS from DNS Resolver but still I see outgoing DNS traffic. :-/

    14 232.701197    192.168.125.2        8.8.4.4              DNS      86    Standard query 0x2d8f PTR 27.120.68.138.in-addr.arpa
        15 233.027563    8.8.4.4              192.168.125.2        DNS      86    Standard query response 0x2d8f Server failure PTR 27.120.68.138.in-addr.arpa
        16 233.040433    192.168.125.2        8.8.4.4              DNS      86    Standard query 0x2d8f PTR 27.120.68.138.in-addr.arpa
        17 233.114054    8.8.4.4              192.168.125.2        DNS      86    Standard query response 0x2d8f Server failure PTR 27.120.68.138.in-addr.arpa
        18 233.341240    192.168.125.2        8.8.4.4              DNS      86    Standard query 0xa56d PTR 39.43.244.104.in-addr.arpa
        19 233.432171    8.8.4.4              192.168.125.2        DNS      86    Standard query response 0xa56d Server failure PTR 39.43.244.104.in-addr.arpa
        20 233.443919    192.168.125.2        8.8.4.4              DNS      86    Standard query 0xa56d PTR 39.43.244.104.in-addr.arpa
        21 233.982914    8.8.4.4              192.168.125.2        DNS      86    Standard query response 0xa56d Server failure PTR 39.43.244.104.in-addr.arpa

    I did the trace with and without promiscuous mode enabled with same result.

    COMMAND: unbound-control -c /var/unbound/unbound.conf lookup .

    The following name servers are used for lookup of .
    forwarding request:
    Delegation with 0 names, of which 0 can be examined to query further addresses.
    It provides 2 IP addresses.
    8.8.8.8         	rto 100 msec, ttl 65, ping 24 var 19 rtt 100, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    8.8.4.4         	rto 407 msec, ttl 425, ping 87 var 80 rtt 407, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    
    

    COMMAND: cat /etc/resolv.conf

    nameserver 127.0.0.1
    search xxxxx.com				; masked my real domain with xxxxx
    nameserver 8.8.8.8
    nameserver 8.8.4.4
    
    

    COMMAND: cat /var/unbound/unbound.conf  (only include a small portion)

    # Interface IP(s) to bind to
    interface: 192.168.120.20
    interface: 2001:470:28:xx::1		; masked my real IP with xx
    interface: 127.0.0.1
    interface: ::1
    
    # Outgoing interfaces to be used
    outgoing-interface: xx.xx.165.51	; masked my real IP with xx
    outgoing-interface: 2001:470:27:xx::2	; masked my real IP with xx
    .
    .
    # Forwarding
    forward-zone:
    	name: "."
    	forward-addr: 8.8.8.8
    	forward-addr: 8.8.4.4
    
    

    Did some tracing and saw the following:  (more detailed logs could be requested if needed)

    Oct 14 00:37:08 pfSense unbound: [78182:2] info: processQueryTargets: 2.android.pool.ntp.org. A IN
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: processQueryTargets: targetqueries 0, currentqueries 0 sentcount 0
    Oct 14 00:37:08 pfSense unbound: [78182:2] info: DelegationPoint<.>: 0 names (0 missing), 2 addrs (0 result, 2 avail) parentNS
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    ip4 8.8.4.4 port 53 (len 16)
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    ip4 8.8.8.8 port 53 (len 16)
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: attempt to get extra 3 targets
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: servselect ip4 8.8.8.8 port 53 (len 16)
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    rtt=155
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: servselect ip4 8.8.4.4 port 53 (len 16)
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug:    rtt=211
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: selrtt 155
    Oct 14 00:37:08 pfSense unbound: [78182:2] info: sending query: 2.android.pool.ntp.org. A IN
    Oct 14 00:37:08 pfSense unbound: [78182:2] debug: sending to target: <.> 8.8.4.4#53 
    








  • @jimp:

    Unlikely to be a bug.

    Check your routing table, and if you have DHCP or PPPoE WANs, make sure to check the box that does not allow them to override your DNS settings. Especially with DHCP, if the ISP sends you a DNS server IP address, dhclient will add a route for it.

    If you are seeing that behavior then somehow the routing table is making the firewall send the traffic that way, usually because of conflicting routes (e.g. received from DHCP, or used as gateway monitor IP addresses, or even plain static route entries)

    As mentioned in my previous post I do not think the routing table is bad and I do not allow any DHCP overide. (not using DHCP. Static on both WANs)
    Please see the attached routingtable.  Routing_ipv4.jpg

    I have masked out some part of the IPs to protect some but you could see the content structure anyway.
    120.0 is the LAN net (where .20 = pfSense)
    121.0 is OpenVPN
    125.0 is a local lan between PFS and a HiLink 4G modem. (that is USBSWITCHED)  WAN2MOBILE
    x.x.165.0 = WAN1
    x.x.50.x remote end of a IPSec