DNS Resolver stopped resolving DNS queries after upgrade to 2.4.4
-
Hello All,
I am always reluctant to run upgrades as about 9/10 times something breaks with anything I ever upgrade, not only pfSense. I bit the bullet and ran a long awaited upgrade to 2.4.4. All clients stop resolving DNS at some point, I have a feeling it happens overnight when the DSL connection breaks for a minute or so.
As a result I've turned off the DNS Resolver and the DNS Server addresses in pfSense's General Setup are assigned automatically with DHCP.
I noticed that rebooting helped once, I think it may have been a cold boot, but not 100% certain now. The reboot today didn't work. Turning off and turning back on DNS Resolver didn't work either.
Here are some system details:
Version 2.4.4-RELEASE (amd64)
built on Thu Sep 20 09:03:12 EDT 2018
FreeBSD 11.2-RELEASE-p3The system is on the latest version.
CPU Type AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: NoHas anyone else noticed this and is there a solution?
Thanks!
-
So what does the resolver log say? What does the system log say when resolver stops resolving? Is it that resolver can not resolve or that its not running or clients can not talk to it..
Have zero issues with the resolver, running it on multiple installs all running 2.4.4, all of which were upgraded from 2.4.3p1
-
[Update]
Just enabled the resolver again, and it is working:
nslookup www.cbc.ca
Server: pfSense.localdomain
Address: 192.168.1.1Non-authoritative answer:
Name: e5220.e12.akamaiedge.net
Address: 72.247.84.53
Aliases: www.cbc.ca
san.cbc.ca.edgekey.net
It will definitely turn itself off at some point, though.
[End Update]I should have thought of the log immediately, thanks. I've highlighted the items that look like they shouldn't be there.
I would have to make it work again and then keep an eye on the logs to see when it stops. Do you happen to see anything out of the ordinary?
Nov 23 04:04:58 named 80637 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use
Nov 23 10:42:16 unbound 89513:0 info: service stopped (unbound 1.8.1).
Nov 23 10:42:16 unbound 89513:0 info: server stats for thread 0: 13 queries, 3 answers from cache, 10 recursions, 0 prefetch, 0 rejected by ip ratelimitingNov 23 11:07:34 named 92296 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use
Nov 23 11:28:17 unbound 38477:0 notice: init module 0: validator
Nov 23 11:28:17 unbound 38477:0 notice: init module 1: iterator
Nov 23 11:28:17 unbound 38477:0 info: start of service (unbound 1.8.1).Nov 23 11:28:39 named 13040 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use
Nov 25 03:44:24 unbound 15778:0 notice: init module 0: validator
Nov 25 03:44:24 unbound 15778:0 notice: init module 1: iterator
Nov 25 03:44:24 unbound 15778:0 info: start of service (unbound 1.8.1).Nov 25 03:44:46 named 42215 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use
Nov 25 10:16:41 unbound 43562:0 notice: init module 0: validator
Nov 25 10:16:41 unbound 43562:0 notice: init module 1: iterator
Nov 25 10:16:41 unbound 43562:0 info: start of service (unbound 1.8.1).Nov 21 21:59:25 named 97520 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use
Nov 2 1 22:01:42 unbound 90591:0 info: service stopped (unbound 1.8.1).
Nov 21 22:01:42 unbound 90591:0 info: server stats for thread 0: 14 queries, 0 answers from cache, 14 recursions, 0 prefetch, 0 rejected by ip ratelimiting0_1543162902267_pfsenseresolverlog.txt
Thank you in advance.
-
@pmrozik whoa you're lucky man
I advice you if you want to use unbound as dns resolver, considering to set no dns servers under System / General setup, left dns servers field blank, and under Services / DNS Resolver, check "Enable DNSSEC" support.
By this way only localhost 127.0.0.1 is effective for dns query and you local network will be smooth and fastest when made dns queryes without hangs.Also you can "force" to use your local dns resolver by adding NAT rule can "grab" all dns traffic and avoid some dns hijacking from your isp.
Tutorial here: https://www.netgate.com/docs/pfsense/dns/redirecting-all-dns-requests-to-pfsense.html -
@pmrozik said in DNS Resolver stopped resolving DNS queries after upgrade to 2.4.4:
127.0.0.1#953: address in use
Your trying to run bind and unbound on the same box? They both want to use 953 for control - so yeah you can have a race condition..
So yeah I could see how you could have issues. Why are you wanting to run bind and unbound at the same time?
If not bind - something else is using 953 since you can not use it.. But I would guess bind.
-
That was the initial problem right after the upgrade. BIND was turned on. I turned it off and it fixed the issue, but what's happening now occurred after BIND was turned off.
So far so good for the last 24 hours, but it's been strange.
-
Thanks.
-
BIND is turned off.
-
I'm not sure what you mean by leaving 127.0.0.1 only for DNS queries? If I leave all the DNS server fields blank, how will it know which other DNS servers to query?
-
DNSSEC support is enabled
Thank you for the link, I'll have a look.
-
-
Unless you have Unbound in forwarding mode, Unbound will never use the DNS server addresses in the System settings. It will go to the root nameservers and resolve the hostname recursively through the various DNS servers involved. So unless you’re using forwarding mode, it’s fine to leave the DNS servers blank (I actually put 127.0.0.1 in one box just to be safe) and let Unbound handle everything.
-
@pmrozik yes, same as @virgiliomi said, outbond work fine without any specific dns servers, and if you want to check, just looking at logfile under
Status/System Logs/System/DNS Resolver
and you see some kind of "Query Response" from dns secure server network, around the world, as confirmation. (Nice uh?) Well, bye. -
Your log shows you had an issue with port 953 as of Nov 25.. So you just did turn off bind then?
As other saying if you are using resolver - that is what it does out of the box then you have zero use for any other NS listed anywhere, nor do you need to pull anything from your isp via dhcp for NS.. So you can turn that off as well.
Unbound in the out of the box resolves.. You do not need to call out any other NS... Pfsense yes out of the box will point to itself ie loopback 127.0.0.1 to ask unbound for what it needs to resolve for example get the package listings, check for update.
To be honest unless you actually understand the difference between forwarder and resolver, and have some specific need your typical user should need to touch these settings at all.. And pfsense will resolve all it needs and all your clients should point to pfsense for dns. Now you will resolve and be using dnssec.. It is for most people optimal configuration... Only if your ISP blocks normal dns, or you have high latency connection should you ever have to forward. Unless you have some desire to leverage some DNS service that is doing some fort of filtering for you, etc.
-
I turned off bind right after the upgrade, so not the 25th.
So far things have been stable, I turned off DNS Forwarding for the DNS Resolver.
The BIND service gets started automatically for some reason, and I don't know why. As soon as I kill the process, DNS resolving comes back.
Pasting logs below:
Nov 27 03:27:40 named 28412 starting BIND 9.11.4-P1 (Extended Support Version) id:2b060b2
Nov 27 03:27:40 named 28412 running on FreeBSD amd64 11.2-RELEASE-p3 FreeBSD 11.2-RELEASE-p3 #17 e6b497fa0a3(RELENG_2_4_4): Thu Sep 20 09:04:45 EDT 2018 root@buildbot3:/crossbuild/ce-244/obj/amd64/WvDslnYb/crossbuild/ce-244/pfSense/tmp/FreeBSD-src/sys/pfSense
Nov 27 03:27:40 named 28412 built with '--localstatedir=/var' '--disable-linux-caps' '--disable-symtable' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--sysconfdir=/usr/local/etc/namedb' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip' '--without-idn' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--without-lmdb' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--enable-threads' '--with-tuning=default' '--without-gssapi' '--with-openssl=/usr' '--disable-native-pkcs11' '--with-dlz-filesystem=yes' '--without-python' '--without-gost' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd11.2' 'build_alias=amd64-portbld-freebsd11.2' 'CC=cc' 'CFLAGS=-O2 -pipe -fstack-protector -isystem /usr/local/include -fno-strict-aliasing' 'LDFLAGS= -fstack-protector' 'LIBS=-L/usr/local/lib'
Nov 27 03:27:40 named 28412 running as: named -c /etc/namedb/named.conf -u bind -t /cf/named/
Nov 27 03:27:40 named 28412 compiled by CLANG 4.2.1 Compatible FreeBSD Clang 6.0.0 (tags/RELEASE_600/final 326565)
Nov 27 03:27:40 named 28412 compiled with OpenSSL version: OpenSSL 1.0.2o-freebsd 27 Mar 2018
Nov 27 03:27:40 named 28412 linked to OpenSSL version: OpenSSL 1.0.2o-freebsd 27 Mar 2018
Nov 27 03:27:40 named 28412 compiled with libxml2 version: 2.9.7
Nov 27 03:27:40 named 28412 linked to libxml2 version: 20907
Nov 27 03:27:40 named 28412 compiled with libjson-c version: 0.13
Nov 27 03:27:40 named 28412 linked to libjson-c version: 0.13
Nov 27 03:27:40 named 28412 compiled with zlib version: 1.2.11
Nov 27 03:27:40 named 28412 linked to zlib version: 1.2.11
Nov 27 03:27:40 named 28412 threads support is enabled
Nov 27 03:27:40 named 28412 ----------------------------------------------------
Nov 27 03:27:40 named 28412 BIND 9 is maintained by Internet Systems Consortium,
Nov 27 03:27:40 named 28412 Inc. (ISC), a non-profit 501(c)(3) public-benefit
Nov 27 03:27:40 named 28412 corporation. Support and training for BIND 9 are
Nov 27 03:27:40 named 28412 available at https://www.isc.org/support
Nov 27 03:27:40 named 28412 ----------------------------------------------------
Nov 27 03:27:40 named 28412 found 2 CPUs, using 2 worker threads
Nov 27 03:27:40 named 28412 using 1 UDP listener per interface
Nov 27 03:27:40 named 28412 using up to 4096 sockets
Nov 27 03:27:40 named 28412 loading configuration from '/etc/namedb/named.conf'
Nov 27 03:27:40 named 28412 unable to open '/usr/local/etc/namedb/bind.keys'; using built-in keys instead
Nov 27 03:27:40 named 28412 using default UDP/IPv4 port range: [49152, 65535]
Nov 27 03:27:40 named 28412 using default UDP/IPv6 port range: [49152, 65535]
Nov 27 03:27:40 named 28412 listening on IPv6 interfaces, port 53
Nov 27 03:27:40 named 28412 could not listen on UDP socket: address in use
Nov 27 03:27:40 named 28412 listening on all IPv6 interfaces failed
Nov 27 03:27:40 named 28412 listening on IPv4 interface bfe0, 192.168.1.1#53
Nov 27 03:27:40 named 28412 listening on IPv4 interface lo0, 127.0.0.1#53
Nov 27 03:27:40 named 28412 listening on IPv4 interface ue0, 192.168.2.104#53
Nov 27 03:27:40 named 28412 listening on IPv4 interface pppoe0, 109.79.226.35#53
Nov 27 03:27:40 named 28412 listening on IPv4 interface ovpns1, 10.0.8.1#53
Nov 27 03:27:40 named 28412 generating session key for dynamic DNS
Nov 27 03:27:40 named 28412 sizing zone task pool based on 0 zones
Nov 27 03:27:40 named 28412 set up managed keys zone for view _default, file 'managed-keys.bind'
Nov 27 03:27:40 named 28412 automatic empty zone: 10.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 16.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 17.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 18.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 19.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 20.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 21.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 22.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 23.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 24.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 25.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 26.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 27.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 28.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 29.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 30.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 31.172.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 168.192.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 64.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 65.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 66.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 67.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 68.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 69.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 70.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 71.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 72.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 73.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 74.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 75.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 76.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 77.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 78.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 79.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 80.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 81.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 82.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 83.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 84.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 85.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 86.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 87.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 88.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 89.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 90.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 91.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 92.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 93.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 94.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 95.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 96.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 97.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 98.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 99.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 100.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 101.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 102.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 103.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 104.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 105.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 106.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 107.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 108.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 109.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 110.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 111.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 112.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 113.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 114.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 115.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 116.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 117.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 118.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 119.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 120.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 121.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 122.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 123.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 124.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 125.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 126.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 127.100.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 0.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 127.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 254.169.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 2.0.192.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 100.51.198.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 113.0.203.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 255.255.255.255.IN-ADDR.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: D.F.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 8.E.F.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 9.E.F.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: A.E.F.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: B.E.F.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: 8.B.D.0.1.0.0.2.IP6.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: EMPTY.AS112.ARPA
Nov 27 03:27:40 named 28412 automatic empty zone: HOME.ARPA
Nov 27 03:27:40 named 28412 /etc/namedb/named.conf:10: couldn't add command channel 127.0.0.1#953: address in use -
@pmrozik said in DNS Resolver stopped resolving DNS queries after upgrade to 2.4.4:
Nov 27 03:27:40 named 28412 starting BIND 9.11.4-P1
It is starting because its enabled.. If your not using it - remove it.. I have it installed but not enabled and it doesn't try and start.. So for whatever reason your install thinks its suppose to start bind even if you have it uncheck in the gui for enabled..
-
Uninstalled it as you suggested.
-
So you could try and reinstall it now and validate it doesn't try and start.. But if your not actually using it - little reason for it to be installed.
Sure you could dig into the xml to see why its trying to start when told not too, etc.
-
You are awesome, thanks a lot for all your help, greatly appreciated.
I probably could reinstall it, but as you said, no need since I'm not using it.
I've definitely learned a couple of things along the way.
