DNS Resolver (unbound) not updating ?



  • Hi;

    I have a pf controllilin 1 WAN and 7 subnets where;
    a) One subnet (only one) uses DNS resolver. Primary to use "Host overides" for some stuff here.
    b) Some subnets uses a Windows AD DNS server. This dns is "DNS server 1" in pf's "general settings".
    c) ...and some subnets uses only ISP DNS servers.

    A computer was "moved" from subnet a) to one of the b) subnets. (Actually, the computer was virtualized, so MAC changed). The windows AD DNS updated accordingly.

    But the a) subnet - with DNS resolver - still gives the old IP back for that computer name, the IP it had when it was a physical machine on the "a)" net.

    It has done this for months, and yes, pfsense has been booted in the meantime.

    Why is this ?

    Note;
    Only setting enabled on DNS Resolver is "Enable DNSSEC support" (apart from a few host overrides).


  • LAYER 8 Global Moderator

    well remove your override then... Or where unbound would be resolving it from.. Did you have unbound set to load dhcp reservations.



  • The computer that moved between subnets, lets refer to it as CLIENT5, is not on the host override list. It has never been. The host overrides are working fine for the subnet.

    That is the question really, exactly where are unbound resolving the CLIENT5 name from ?
    The name of CLIENT5 is set to another IP in the DNS configurated as "DNS server 1" in pf's "general settings". (On the subnets not using unbound the name resolves correctly.)
    The name of CLIENT5 does not exist in the ISP's DNS server (the only other DNS's mentioned on pf)

    No, "Register DHCP leases in the DNS Resolver" is not set. "Register DHCP static mappings in the DNS Resolver" are also unchecked.
    (Note:, the name of CLIENT5 exist in the DHCP leases list - with the correct IP.)

    To me it seems like unbound picked up the name and IP combo for CLIENT5 a long time ago (back when it actually was correct) and has never updated itself since. So it keeps answering CLIENT5 name request with the wrong IP. Any ideas why ?


  • LAYER 8 Global Moderator

    unbound doesn't pick up anything.. It resolves..

    Entries are only the cache for the length of the ttl on the record.. A restart of unbound would flush the cache.

    Please post exactly what is happening when you query the specific FQDN from unbound... client5 is not a fqdn, and will not resolve.. But you client could be adding its own search suffix and resolving that from??

    It is returning rfc1918 space right?

    Where does unbound say it would lookup the fqdn ?
    /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

    What is in the local data, dump it and check - for example here I look for local data of fqdn of box of mine called pi.local.lan

    [2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf list_local_data | grep pi.local.lan 
    pi.local.lan.   3600    IN      A       192.168.3.31
    31.3.168.192.in-addr.arpa.      3600    IN      PTR     pi.local.lan.
    [2.4.4-RELEASE][admin@sg4860.local.lan]/:  
    


  • Result of unbound-control (Note: i masked the real FQDN with x's in the snippet below):

    The following name servers are used for lookup of xxxxx.xxx.xx.
    The noprime stub servers are used:
    Delegation with 0 names, of which 0 can be examined to query further addresses.
    It provides 1 IP addresses.
    10.100.102.11   	NoDNSSEC rto 12 msec, ttl 121, ping 0 var 3 rtt 50, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    

    ... the IP returned/mentioned above is the wrong one (it has been in use by this FQDN previously, but thats a long time ago).


  • LAYER 8 Global Moderator

    noprime stub servers..

    so you did a lookup for host.domain.tld and you got back the NS its going to ask.. This 10.100.102.11 box..

    For example if I ask unbound what will it use to lookup www.netgate.com

    [2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup www.netgate.com
    The following name servers are used for lookup of www.netgate.com.
    ;rrset 2982 2 0 7 3
    netgate.com.    2982    IN      NS      ns1.netgate.com.
    netgate.com.    2982    IN      NS      ns2.netgate.com.
    ;rrset 2982 1 0 8 3
    ns2.netgate.com.        2982    IN      A       162.208.119.38
    ;rrset 2982 1 0 3 3
    ns1.netgate.com.        2982    IN      A       208.123.73.80
    ;rrset 2982 1 0 3 3
    ns1.netgate.com.        2982    IN      AAAA    2610:160:11:11::80
    Delegation with 2 names, of which 1 can be examined to query further addresses.
    It provides 3 IP addresses.
    2610:160:11:11::80      rto 339 msec, ttl 282, ping 15 var 81 rtt 339, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    208.123.73.80           rto 226 msec, ttl 282, ping 14 var 53 rtt 226, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    162.208.119.38          rto 338 msec, ttl 282, ping 6 var 83 rtt 338, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    [2.4.4-RELEASE][admin@sg4860.local.lan]/: 
    

    Its says it will use NS 2610, 208 and 162 that will provide the answer.

    That is not the answer... So your unbound is asking the 10.100.102.11 box for that fqdn.. So it is the NS that is returning the wrong answer.

    You have a domain override setup for whatever domain your asking about or your forwarding?

    Look at your unbound.conf file - PM it to me and can take a look see.
    example here is mine

    [2.4.4-RELEASE][admin@sg4860.local.lan]/: cat /var/unbound/unbound.conf 
    ##########################
    # Unbound Configuration
    ##########################
    
    ##
    # Server configuration
    ##
    server:
    
    chroot: /var/unbound
    username: "unbound"
    directory: "/var/unbound"
    pidfile: "/var/run/unbound.pid"
    use-syslog: yes
    port: 53
    verbosity: 3
    hide-identity: no
    hide-version: no
    harden-glue: yes
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
    do-daemonize: yes
    module-config: "validator iterator"
    unwanted-reply-threshold: 0
    num-queries-per-thread: 512
    jostle-timeout: 200
    infra-host-ttl: 900
    infra-cache-numhosts: 50000
    outgoing-num-tcp: 20
    incoming-num-tcp: 20
    edns-buffer-size: 4096
    cache-max-ttl: 86400
    cache-min-ttl: 3600
    harden-dnssec-stripped: yes
    msg-cache-size: 20m
    rrset-cache-size: 40m
    
    num-threads: 4
    msg-cache-slabs: 4
    rrset-cache-slabs: 4
    infra-cache-slabs: 4
    key-cache-slabs: 4
    outgoing-range: 4096
    #so-rcvbuf: 4m
    auto-trust-anchor-file: /var/unbound/root.key
    prefetch: yes
    prefetch-key: yes
    use-caps-for-id: yes
    serve-expired: yes
    # Statistics
    # Unbound Statistics
    statistics-interval: 0
    extended-statistics: yes
    statistics-cumulative: yes
    
    # SSL Configuration
    
    # Interface IP(s) to bind to
    interface: 192.168.3.253
    interface: 2001:470:snipped::253
    interface: 192.168.9.253
    interface: 192.168.2.253
    interface: 192.168.6.253
    interface: 192.168.4.253
    interface: 192.168.7.253
    interface: 127.0.0.1
    interface: ::1
    
    # Outgoing interfaces to be used
    outgoing-interface: 2001:470:snipped::2
    outgoing-interface: 64.53.snipped
    
    # DNS Rebinding
    # For DNS Rebinding prevention
    private-address: 10.0.0.0/8
    private-address: ::ffff:a00:0/104
    private-address: 172.16.0.0/12
    private-address: ::ffff:ac10:0/108
    private-address: 169.254.0.0/16
    private-address: ::ffff:a9fe:0/112
    private-address: 192.168.0.0/16
    private-address: ::ffff:c0a8:0/112
    private-address: fd00::/8
    private-address: fe80::/10
    # Set private domains in case authoritative name server returns a Private IP address
    private-domain: "home.lan"
    domain-insecure: "home.lan"
    
    
    # Access lists
    include: /var/unbound/access_lists.conf
    
    # Static host entries
    include: /var/unbound/host_entries.conf
    
    # dhcp lease entries
    include: /var/unbound/dhcpleases_entries.conf
    
    
    
    # Domain overrides
    include: /var/unbound/domainoverrides.conf
    
    
    # Unbound custom options
    server:
    private-domain: "plex.direct"
    so-reuseport: no
    log-queries: yes
    log-replies: yes
    #private-address: ::/0  # filters out all AAAA !
    
    
    ###
    # Remote Control Config
    ###
    include: /var/unbound/remotecontrol.conf
    [2.4.4-RELEASE][admin@sg4860.local.lan]/: 
    

    I only snipped out my public interface IPs



  • Wow, You are fast!

    And you are correct, of course, the IP 10.100.102.11 mentioned is to the DNS server (DNS on a windows AD). Its NOT the IP returned when the CLIENT5 name are resolved. My mistake.

    (note: When doing unbound-control with www.netgate.com I get beck pretty much the same as You).

    So it seems the problem is somewhere within the windows AD DNS then ?

    The AD DNS somehow resolves the FQDN of CLIENT5 to be 10.100.12.8 when "going through" unbound, but the correct 10.100.105.79 when it set (by DHCP) to be used as the primary DNS ... ?


  • LAYER 8 Global Moderator

    Do you have views setup on your AD setup, or conditional forwarders, etc.

    Without more info to your setup of your dns in AD, not sure.. What version of windows you running 2k12, 2k16, 2003? etc.. Over the years they have made changes to how DNS works in windows for features... Your not still using wins are you?

    Are you syncing DNS across multiple servers in your AD.. I assume you have it directory-integrated, etc.

    to be used as the primary DNS ... ?

    Not a fan when users state such a thing... While you can point clients to more than 1 NS.. you have to be SURE!!!! that these NS are going to resolve/forward and return the same info..

    For example - you can not point a client to your local AD dns, while also listing say 8.8.8.8 be it 2nd NS or not.. You will have a bad day if you setup this way.. Since you can never be sure where a client will ask if there is more than 1 NS listed. So sure you could point to multiple AD dns that all return the same info.. But you can not point a client to say

    NS1 10.x.x.x
    NS2 8.8.8.8

    Without running into problems at some point..



  • The dc used here (10.100.102.11) are 2012 R2 std. Not wins, no. There exist "backup" dc's. It's a single site domain, no forest's or other "advanced" stuff used.

    I have often used multiple DNS's in DHCP, but I see your point.

    However, to clearify a bit (Note: i masked the real FQDN's with x's in the snippet below);

    When on a windows client in the "outbound subnet", the DNS settings are as follows;

    [D:\Source\Code\phy]ipconfig /all
    
    Windows IP Configuration
       ...
       DNS Servers . . . . . . . . . . . : 10.100.0.1
                                           10.100.102.11
       ...
    

    The 10.100.0.1 is the pfSense, DNS resolver enabled here (to enable som "host overrides"). So we ask to resolve;

    [D:\Source\Code\phy]nslookup client5.xxx.xx
    Server:  pfSense.xxx.xx
    Address:  10.100.0.1
    
    Name:    client5.xxx.xx
    Address:  10.100.12.8
    

    Now, earlier we established that outbound on pfSense are actually using 10.100.102.11

    The following name servers are used for lookup of client5.xxx.xx.
    The noprime stub servers are used:
    Delegation with 0 names, of which 0 can be examined to query further addresses.
    It provides 1 IP addresses.
    10.100.102.11   	NoDNSSEC rto 12 msec, ttl 366, ping 0 var 3 rtt 50, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    

    But when we resolve through 10.100.102.11 directly;

    [D:\Source\Code\phy]nslookup client5.xxx.xx 10.100.102.11
    Server:  pdc03.xxx.xx
    Address:  10.100.102.11
    
    Name:    client5.xxx.xx
    Address:  10.100.105.79
    

    We get another IP. This is the correct one.

    I did a quick check on the DNS setup on 10.100.102.11: exporting everything to a .txt file and the a search, but 10.100.12.8 was not found anywhere.


  • LAYER 8 Global Moderator

    @Lumberjack said in DNS Resolver (unbound) not updating ?:

    Windows IP Configuration
    ...
    DNS Servers . . . . . . . . . . . : 10.100.0.1
    10.100.102.11
    ...

    That is going to be problematic..

    Your clients should point to AD if they are members of the AD or need to resolve stuff in AD... You would then forward from your AD to pfsense if you want AD to resolve stuff in pfsense overrides... But why would you not just put the overrides in AD in the first place?

    Where are you doing that direct query from? Same subnet as pfsense does it query to your AD?

    Just turn on logging in unbound and you can see exactly where your answer coming from..

    server:
    log-queries: yes
    log-replies: yes

    Put that in the custom options box on unbound setup..

    Then set logging to level 3 or higher

    Then query pfsense for this client and you can see exactly where it comes from... So I have a domain override setup for testing to windows dns... for home.lan 192.168.2.220

    So when I query pfsense for host.home.lan it goes and asks the windows dns.. You can see exactly what unbound is doing with logging turned on.

    Mar 21 10:30:23 	unbound 	6448:1 	info: 192.168.9.100 host.home.lan. A IN NOERROR 0.759713 0 58
    Mar 21 10:30:23 	unbound 	6448:1 	info: query response was ANSWER
    Mar 21 10:30:23 	unbound 	6448:1 	info: reply from <home.lan.> 192.168.2.220#53
    Mar 21 10:30:23 	unbound 	6448:1 	info: response for host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: iterator operate: query host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	debug: sending to target: <home.lan.> 192.168.2.220#53
    Mar 21 10:30:23 	unbound 	6448:1 	info: sending query: host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: processQueryTargets: host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: iterator operate: query host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	debug: sending to target: <home.lan.> 192.168.2.220#53
    Mar 21 10:30:23 	unbound 	6448:1 	info: sending query: host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: processQueryTargets: host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: resolving host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: validator operate: query host.home.lan. A IN
    Mar 21 10:30:23 	unbound 	6448:1 	info: 192.168.9.100 host.home.lan. A IN 
    

    You can see my client from 192.168.9.100 ask.. Unbound goes and ask 192.168.2.220 and gets answer

    You see what my client gets back.

    $ dig @192.168.9.253 host.home.lan
    
    ; <<>> DiG 9.12.3-P1 <<>> @192.168.9.253 host.home.lan
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12831
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ;; QUESTION SECTION:
    ;host.home.lan.                 IN      A
    
    ;; ANSWER SECTION:
    host.home.lan.          3600    IN      A       10.10.10.10
    
    ;; Query time: 760 msec
    ;; SERVER: 192.168.9.253#53(192.168.9.253)
    ;; WHEN: Thu Mar 21 10:30:23 Central Daylight Time 2019
    ;; MSG SIZE  rcvd: 58
    

    When I ask unbound where it would lookup host.home.lan I get

    [2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup host.home.lan
    The following name servers are used for lookup of host.home.lan.
    forwarding request:
    Delegation with 0 names, of which 0 can be examined to query further addresses.
    It provides 1 IP addresses.
    192.168.2.220           rto 284 msec, ttl 558, ping 0 var 71 rtt 284, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
    [2.4.4-RELEASE][admin@sg4860.local.lan]/: 
    

    Why when you did for your client you see noprime stub? Please post your unbound config.. Do you have unbound set as stub? vs default transparent You cearly have something more than the gui configurations for unbound going on when it says your using stub zone and not prime.

    You are not just forwarding to your AD to lookup specific domain... You or it would say forwarding like mine above when setup domain override to go ask windows dns at 192.168.2.220 for home.lan



  • Detected that the site currently is running pfsense 2.3-RELEASE (amd64), FreeBSD 10.3-RELEASE. There is no 'log_replies' unbound option in that freebsd version.

    I have scheduled a upgrade. Reporting back after that.

    And I detected that unbound is set with "domain override" of the domain (eg. xxx.xx of the client5.xxx.xx). This possibly explains missing calls to prime zone's.


  • LAYER 8 Global Moderator

    2.3?? Dude that is no longer supported, end of life... You need to upgrade to current..

    Let us know after the upgrade.. But its still asking your AD for that..

    What version of unbound is it running btw?



  • Ok, pf on 2.4.4 now.

    I believe I have found source of the the problem. it's this:
    https://forum.netgate.com/topic/135409/unbound-dhcp-registration-checkbox-bug

    First, I enabled the unbound log (as suggested) an this was the result:

    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. AAAA IN NOERROR 0.000000 1 31 
    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. AAAA IN 
    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. A IN NOERROR 0.000000 1 47 
    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. A IN 
    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 1.0.100.10.in-addr.arpa. PTR IN NOERROR 0.000000 1 69 
    Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 1.0.100.10.in-addr.arpa. PTR IN
    

    That's all. No contact with AD DNS.

    So, after some thinkering, i did a cat of the unbound config and found this ;

    ...
    # dhcp lease entries
    include: /var/unbound/dhcpleases_entries.conf
    ...
    

    ...whut ?! thats not even enabled?

    Then I did a cat on dhcpleases_entries.conf and, yes, i found client5.xxx.xx listed.
    ...along with approx other 10 names registered, where only 2 still exists today.

    Somebody has probably, a long time ago, enabled/checked "DHCP Registration" in the pf GUI. Then unchecked it again a short time after enabling it. But the names collected got stuck in unbound. Not good.

    Should I just truncate the dhcpleases_entries.conf or do there exist a more proper way to get rid of it ?


  • LAYER 8 Global Moderator

    Just delete all the entries in there should be fine.



  • Done. It worked, the correct IP are returned now.

    One remark;

    Something for the devs to consider, This;

    usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

    ... should really give some kind of indication that the name in question do exist in a local table.



  • @Lumberjack said in DNS Resolver (unbound) not updating ?:

    Something for the devs to consider, This;

    usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

    ... should really give some kind of indication that the name in question do exist in a local table.

    Talk to the unbound devs then: https://nlnetlabs.nl



  • @Lumberjack said in DNS Resolver (unbound) not updating ?:

    ...whut ?! thats not even enabled?

    Hummm. This rings a bell to me.
    An issue that was solved a couple of updates ago ... emptying the dhcp leases file when dhcpleases process is shut down.

    You 'missed' that upgrade I guess ^^


Log in to reply