Unable to start Unbound after switch away from KEA
-
After an ugprade to 24.11 I decided to give KEA a try. I since then switched back to ISC and thought nothing of it.
Checking my monitoring, I noticed that Unbound is not started - DNS failures. I tried to start Unbound and received the following error
This is a test SG-1100 unit.
-
Hmm, what do you see on line 17 of /var/unbound/unbound.conf ?
I can't think of anything that would apply there but....
-
@stephenw10 said in Unable to start Unbound after switch away from KEA:
l
cat /var/unbound/unbound.conf | nl 1 ########################## 2 # Unbound Configuration 3 ########################## 4 ## 5 # Server configuration 6 ## 7 server: 8 chroot: /var/unbound 9 username: "unbound" 10 directory: "/var/unbound" 11 pidfile: "/var/run/unbound.pid" 12 use-syslog: yes 13 port: 53 14 verbosity: 15 hide-identity: yes 16 hide-version: yes 17 harden-glue: yes 18 do-ip4: yes 19 do-ip6: yes 20 do-udp: yes 21 do-tcp: yes 22 do-daemonize: yes 23 module-config: "validator iterator" 24 unwanted-reply-threshold: 0 25 num-queries-per-thread: 4096 26 jostle-timeout: 200 27 infra-keep-probing: yes 28 infra-host-ttl: 900 29 infra-cache-numhosts: 10000 30 outgoing-num-tcp: 31 incoming-num-tcp: 32 edns-buffer-size: 1332 33 cache-max-ttl: 86400 34 cache-min-ttl: 0 35 harden-dnssec-stripped: yes 36 msg-cache-size: 4m 37 rrset-cache-size: 8m 38 num-threads: 2 39 msg-cache-slabs: 2 40 rrset-cache-slabs: 2 41 infra-cache-slabs: 2 42 key-cache-slabs: 2 43 outgoing-range: 4096 44 so-rcvbuf: 12m 45 auto-trust-anchor-file: /var/unbound/root.key 46 prefetch: no 47 prefetch-key: no 48 use-caps-for-id: no 49 serve-expired: no 50 sock-queue-timeout: 0 51 aggressive-nsec: no 52 # Statistics 53 # Unbound Statistics 54 statistics-interval: 0 55 extended-statistics: yes 56 statistics-cumulative: yes 57 # TLS Configuration 58 tls-cert-bundle: "/etc/ssl/cert.pem" 59 # Interface IP addresses to bind to 60 interface-automatic: yes 61 # DNS Rebinding 62 # Access lists 63 include: /var/unbound/access_lists.conf 64 # Static host entries 65 include: /var/unbound/host_entries.conf 66 # dhcp lease entries 67 include: /var/unbound/dhcpleases_entries.conf 68 # Domain overrides 69 include: /var/unbound/domainoverrides.conf 70 # Forwarding 71 forward-zone: 72 name: "." 73 forward-addr: 9.9.9.9 74 forward-addr: 149.112.112.112 75 ### 76 # Remote Control Config 77 ### 78 include: /var/unbound/remotecontrol.conf -
Took a hail mary and ran the command
[24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root: /usr/local/sbin/unbound
[24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root:Unbound started without issues.........
No clue what is going on.
-
@michmoor said in Unable to start Unbound after switch away from KEA:
[24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root: /usr/local/sbin/unbound
That shouldn't work.
To name one : unbound, by default, uses classic freebsd file paths like /usr/local/etc/unbound/unbound.conf to start up.[24.11-RELEASE][root@pfSense.bhf.tld]/root: unbound -h usage: unbound [options] start unbound daemon DNS resolver. -h this help. -c file config file to read instead of /usr/local/etc/unbound/unbound.conf file format is described in unbound.conf(5). -d do not fork into the background. -p do not create a pidfile. -v verbose (more times to increase verbosity). -V show version number and build options. Version 1.22.0 BSD licensed, see LICENSE in source package for details. Report bugs to unbound-bugs@nlnetlabs.nl or https://github.com/NLnetLabs/unbound/issuesThere is a config file at /usr/local/etc/unbound/ but that one is a generic one, enumerating all the default config values, and it won't work in the pfSEnse environment.
The actual unbound config file isn't there, it's here : /var/unbound/unbound.confA manual unbound startup would look like
unbound -c /var/unbound/unbound.confThat said, I"'m using 24.11 since yesterday, maybe things changed ?
-
Nope it's using that conf file from the custom location and failing for some reason.
The first thing I would do is just resave the DNS Resolver settings to regenerate that conf file. Though nothing in that current file looks obviously like a problem to me.
-
Starting unbound the way i did did allow the service to show green in the pfsense GUI but no DNS resolution was occurring
ps auxx | grep unbound root 73481 0.0 0.2 12860 1936 - Is 19:55 0:00.02 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d moore.lan -p /var/run/unbound.pid -u /var/unbound/d unbound 87474 0.0 0.7 40588 6956 - Is Thu23 0:18.40 /usr/local/sbin/unbound root 27 0.0 0.2 12952 2064 0 S+ 19:58 0:00.01 grep unbound [24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root: kill -9 87474 [24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root: ps auxx | grep unbound root 73481 0.0 0.2 12860 1936 - Is 19:55 0:00.02 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d moore.lan -p /var/run/unbound.pid -u /var/unbound/dh root 2975 0.0 0.2 12952 2064 0 S+ 19:59 0:00.01 grep unbound [24.11-RELEASE][admin@nyc-fw1-inet.moore.lan]/root:I have re-saved the unbound configuration again, apply changes , and still the service remains down with the following error seen in the log
-
Manually starting the service
unbound -c /var/unbound/unbound.conf /var/unbound/unbound.conf:17: error: number expected /var/unbound/unbound.conf:17: error: unknown keyword 'yes' /var/unbound/unbound.conf:33: error: number expected read /var/unbound/unbound.conf failed: 3 errors in configuration file [1733101412] unbound[3798:0] fatal error: Could not read config file: /var/unbound/unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-checkconfHere is the unbound.conf again.
cat /var/unbound/unbound.conf ########################## # Unbound Configuration ########################## ## # Server configuration ## server: chroot: /var/unbound username: "unbound" directory: "/var/unbound" pidfile: "/var/run/unbound.pid" use-syslog: yes port: 53 verbosity: hide-identity: yes hide-version: yes harden-glue: yes do-ip4: yes do-ip6: yes do-udp: yes do-tcp: yes do-daemonize: yes module-config: "validator iterator" unwanted-reply-threshold: 0 num-queries-per-thread: 4096 jostle-timeout: 200 infra-keep-probing: yes infra-host-ttl: 900 infra-cache-numhosts: 10000 outgoing-num-tcp: incoming-num-tcp: edns-buffer-size: 1332 cache-max-ttl: 86400 cache-min-ttl: 0 harden-dnssec-stripped: yes msg-cache-size: 4m rrset-cache-size: 8m num-threads: 2 msg-cache-slabs: 2 rrset-cache-slabs: 2 infra-cache-slabs: 2 key-cache-slabs: 2 outgoing-range: 4096 so-rcvbuf: 12m auto-trust-anchor-file: /var/unbound/root.key prefetch: no prefetch-key: no use-caps-for-id: no serve-expired: no sock-queue-timeout: 0 aggressive-nsec: no # Statistics # Unbound Statistics statistics-interval: 0 extended-statistics: yes statistics-cumulative: yes # TLS Configuration tls-cert-bundle: "/etc/ssl/cert.pem" # Interface IP addresses to bind to interface-automatic: yes # DNS Rebinding # Access lists include: /var/unbound/access_lists.conf # Static host entries include: /var/unbound/host_entries.conf # dhcp lease entries include: /var/unbound/dhcpleases_entries.conf # Domain overrides include: /var/unbound/domainoverrides.conf # Forwarding forward-zone: name: "." forward-addr: 9.9.9.9 forward-addr: 149.112.112.112 ### # Remote Control Config ### include: /var/unbound/remotecontrol.conf -
Counting the lines in the conf file, Line 17 is the following
hide-version: yesI went into the GUI, DNS Resolver , Advanced Settings, and unchecked Hide Version.
Apply changes.
DNS Resolver starts without issues and name resolution finally works.@stephenw10 what is that setting and why did having it checked break Unbound? Seems like a redmine issue but i don't know what the fault is.
Checking my 6100 HighAvailability pair, they have it enabled and no Unbound issues post upgrade. Even did a failover to test and it works.Any guidance you can offer i would greatly appreciate it!
-
This one :
runs because you've set :
If it is possible : disable that option.
dhcpdleases restarts unbound every time a new DHCP lease event is received.
This can give you the impression that 'unbound' wasn't running as it was 'restarting again (and again, and again etc etc).
Unbound restarts do take some time, several seconds, or more.You see the command line option of that dhcpdleases process ?
-p /var/run/unbound.pidwhich means that when you restart manually unbound, the PID changes.
dhcpdleases also has to be restarted - not sure if you can do this manually. -
@Gertjan Thanks Gertjan but you are misunderstanding the problem.
DNS resolver wasn't running due to a parameter set it did not recognize or like for some reason. After negating that in the advanced options in Unbound, it started up successfully and resolved hostnames.
The issue here is why was it broken in the first place. The only connection i have to KEA is that it was the last change i made on the system before i reverted back.
-
Line 17 is actually
hide-identity: yes. But that's not the issue that's a default setting. The problem is that you had no log verbosity value on the line before. It's expecting a number for the log verbosity and sees not-a-number at the start of line 17.Resaving the Unbound Advanced page will have added that value back.
The same applies to the values for:
outgoing-num-tcp: incoming-num-tcp:However I have no idea what might have caused those to not be populated. Unbound should never have allowed those empty values.
-
@stephenw10 I gotcha.
As mentioned, the last change I made was the KEA switchover. I don't know if KEA is involved with Unbound in this case, but that was the last change I made. I don't understand what could suddenly cause Unbound to find a problem with the conf file. Outside of configuring Unbound via the GUI as far as i know there is no direct way to modify the configuration and have it saved. I could be wrong of course. -
Indeed. I can see why it rejected that conf file because there are values missing. What I can't understand is how that was ever generated and passed the Unbound tests.
If you revert that change to the Unbound settings I imagine it will still start no problem.
-
@stephenw10 I reverted my change in Unbound and here is the conf file now.
########################## # Unbound Configuration ########################## ## # Server configuration ## server: chroot: /var/unbound username: "unbound" directory: "/var/unbound" pidfile: "/var/run/unbound.pid" use-syslog: yes port: 53 verbosity: 1 hide-identity: yes hide-version: yes harden-glue: yes do-ip4: yes do-ip6: yes do-udp: yes do-tcp: yes do-daemonize: yes module-config: "validator iterator" unwanted-reply-threshold: 0 num-queries-per-thread: 512 jostle-timeout: 200 infra-keep-probing: yes infra-host-ttl: 900 infra-cache-numhosts: 10000 outgoing-num-tcp: 10 incoming-num-tcp: 10 edns-buffer-size: 1332 cache-max-ttl: 86400 cache-min-ttl: 0 harden-dnssec-stripped: yes msg-cache-size: 4m rrset-cache-size: 8m num-threads: 2 msg-cache-slabs: 2 rrset-cache-slabs: 2 infra-cache-slabs: 2 key-cache-slabs: 2 outgoing-range: 4096 so-rcvbuf: 12m auto-trust-anchor-file: /var/unbound/root.key prefetch: no prefetch-key: no use-caps-for-id: no serve-expired: no sock-queue-timeout: 0 aggressive-nsec: no # Statistics # Unbound Statistics statistics-interval: 0 extended-statistics: yes statistics-cumulative: yes # TLS Configuration tls-cert-bundle: "/etc/ssl/cert.pem" # Interface IP addresses to bind to interface-automatic: yes # DNS Rebinding # Access lists include: /var/unbound/access_lists.conf # Static host entries include: /var/unbound/host_entries.conf # dhcp lease entries include: /var/unbound/dhcpleases_entries.conf # Domain overrides include: /var/unbound/domainoverrides.conf # Forwarding forward-zone: name: "." forward-addr: 9.9.9.9 forward-addr: 149.112.112.112 ### # Remote Control Config ### include: /var/unbound/remotecontrol.conf -
Yup so now those 3 values are all populated. And I imagine it starts fine?
-
Yep starts up fine, no issues.
Unless i can reproduce it or others have a similar issue might have to throw this in the mystery box.
