[sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100

[javen]

I have a Netgate sg3100 with 32GB eMMC running pfSense 21.05.2 and have experienced this issue since June 2020 when I bought it.

After performing a factory reset and configuring the startup wizard, the router works as expected. However, after I reboot my router from the initial configuration, I am unable to ping google.com from my computer.

I believe this is an issue with the unbound DNS Resolver, which appears to be running in the Status > Services section, but not running correctly. If I restart the unbound service, DNS works again and I can ping google.com from my computer.

I don't want to manually log in to the router to restart the unbound service if the router gets power cycled, since I may not be at home to reset it for my family if the power is out longer than my UPS's battery can last.

For now, I have disabled the DNS Resolver and enabled the DNS Forwarder. This config works after rebooting the sg3100.

However, I'd like to get the DNS Resolver working, since caching name servers locally should reduce page load times and also offer better privacy than always fetching DNS records from third party DNS servers.

First question: is this problem unique to my hardware, all Netgate sg3100's, all Netgate arm appliances, or is this an issue with pfsense on any appliance?

Second question: What is the best solution for getting the DNS Resolver working without restarting unbound after a system reboot or power cycle?

Thanks for the help!

System Logs / System / DNS Resolver (oldest logs at the top, newest at the bottom; I measured this on Jan 20, but the log timestamp is Dec 27 when the sg3100 isn't able to reach an external time server; seems a bit odd that the system time would jump around like this, but whatever).

I grabbed a little extra log history before disabling the DNS Forwarder, enabling the DNS Resolver, and rebooting the sg3100.

Jan 20 22:37:36 	dnsmasq 	1390 	reading /etc/resolv.conf
Jan 20 22:37:36 	dnsmasq 	1390 	using nameserver 1.1.1.1#53
Jan 20 22:37:36 	dnsmasq 	1390 	using nameserver 8.8.8.8#53
Jan 20 22:37:36 	dnsmasq 	1390 	read /etc/hosts - 3 addresses
Jan 20 22:37:40 	dnsmasq 	1390 	reading /etc/resolv.conf
Jan 20 22:37:40 	dnsmasq 	1390 	ignoring nameserver 127.0.0.1 - local interface
Jan 20 22:37:40 	dnsmasq 	1390 	using nameserver 1.1.1.1#53
Jan 20 22:37:40 	dnsmasq 	1390 	using nameserver 8.8.8.8#53
Dec 27 11:08:07 	dnsmasq 	38096 	started, version 2.85 cachesize 10000
Dec 27 11:08:07 	dnsmasq 	38096 	compile time options: IPv6 GNU-getopt no-DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-cryptohash no-DNSSEC loop-detect no-inotify dumpfile
Dec 27 11:08:07 	dnsmasq 	38096 	reading /etc/resolv.conf
Dec 27 11:08:07 	dnsmasq 	38096 	ignoring nameserver 127.0.0.1 - local interface
Dec 27 11:08:07 	dnsmasq 	38096 	using nameserver 1.1.1.1#53
Dec 27 11:08:07 	dnsmasq 	38096 	using nameserver 8.8.8.8#53
Dec 27 11:08:07 	dnsmasq 	38096 	read /etc/hosts - 3 addresses
Jan 20 23:09:31 	dnsmasq 	38096 	exiting on receipt of SIGTERM
Jan 20 23:09:55 	unbound 	61737 	[61737:0] notice: init module 0: validator
Jan 20 23:09:55 	unbound 	61737 	[61737:0] notice: init module 1: iterator

After rebooting the sg3100:

Jan 20 23:09:55 	unbound 	61737 	[61737:0] info: start of service (unbound 1.12.0).
Dec 27 11:08:20 	unbound 	61049 	[61049:0] notice: init module 0: validator
Dec 27 11:08:20 	unbound 	61049 	[61049:0] notice: init module 1: iterator
Dec 27 11:08:20 	unbound 	61049 	[61049:0] info: start of service (unbound 1.12.0).
Dec 27 11:08:21 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:21 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:21 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:21 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:21 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:22 	unbound 	61049 	[61049:1] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:23 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:23 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:23 	unbound 	61049 	[61049:0] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:23 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:23 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:23 	unbound 	61049 	[61049:1] info: generate keytag query _ta-4f66. NULL IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN
Dec 27 11:08:24 	unbound 	61049 	[61049:1] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN 

After restarting the unbound service:

Jan 20 23:22:37 	unbound 	53993 	[53993:0] notice: init module 0: validator
Jan 20 23:22:37 	unbound 	53993 	[53993:0] notice: init module 1: iterator
Jan 20 23:22:37 	unbound 	53993 	[53993:0] info: start of service (unbound 1.12.0).
Jan 20 23:22:38 	unbound 	53993 	[53993:0] info: generate keytag query _ta-4f66. NULL IN 

[keyser]

@javen That is definitely not the normal. The unbound resolver should start automatically like any other service on the box.

What packages are installed on your box? PfBlockerNG (devel) perhaps? That interacts quite a lot with Unbound, and can cause issues if it not configured correctly.

If you have upgraded through pfSense versions you might be better of taking a config backup, and then get the restore image for the SG-3100 from support and really “reset” it :-) The facorty reset in the Shell/GUI only removes your config files and currently installed packages. If - through versions/upgrades or powerfailures - you have had system service files for unbound added or modified or corrupted, they will not be cleaned unless you get the restore image from support.

[javen]

@keyser said in [sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100:

That is definitely not the normal. The unbound resolver should start automatically like any other service on the box.
What packages are installed on your box?

I have had pfBlockerNG installed in the past before I figured out that unbound was the problem and restarting the service after every reboot fixed the problem.
Currently I have no packages installed. I ran Diagnostics > Factory Defaults > Factory Reset from the GUI.

restore image for the SG-3100 from support

I've re-imaged the sg3100 a few times in the past and it didn't resolve the issue. Once in June 2020 shortly after getting it, and again in August 2020 using pfSense-netgate-SG-3100-recover-2.4.5-RELEASE-p1-armv6.img.gz
I haven't re-imaged since pfSense Plus.

I'll re-image it one more time and see if the issue persists.

[javen]

Also, searching for failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN pointed me to a Netgate Forums post by gjaltemba on 2015-03-20 with the title "Failed to prime trust anchor".

For which the solution was to uncheck Services > DNS Resolver > General Settings > General DNS Resolver Options > DNSSEC.

Disabling the DNS Resolver's DNSSEC support works, and the DNS Resolver seems to run correctly after rebooting the router. No manual unbound service restart required. Getting closer...

[javen]

Welp, I guess I get to re-flash the SG-3100 a little sooner than planned.

After enabling a few more settings in the DNS Resolver, I'm now no longer able to reach the SG-3100 on my network.

DNS Resolver
	DNSSEC
		Enable DNSSEC Support = unchecked (must be unchecked)
	DNS Query Forwarding
		Enable Forwarding Mode = checked (doesn't matter)

Then I checked these are rebooted:

DNS Resolver
	DNS Query Forwarding
		Use SSL/TLS for outgoing DNS Queries to Forwarding Servers = checked

	DHCP Registration
		Register DHCP leases in the DNS Resolver = checked
	Static DHCP
		Register DHCP static mappings in the DNS Resolver = checked

Enabling the last 3 settings made the router inaccessible. :(

[keyser]

@javen The “Register DHCP leases in DNS” must be unchecked in the current implementation of Unbound in pfSense. This has been a long standing issue…
The static DHCP clients registration is not an issue

Otherwise the DNS resolver will reload every time a DHCP client gets an IP. This could very likely be your problem at startup. If you have a bunch af clients getting an IP when the DHCP server is up, it could likely prevent Unbound from actually starting for quite a while because of sustained reloads.

[Gertjan]

This :

@javen said in [sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100:

failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN

is surely related to this :

the system was thrown back in the past 4 weeks.

For DNSSEC to work correctly, the time on your pfSense must be correct. Ok if it is out of sync a second or so. 4 weeks is a disaster.

What happens if you :
Make a backup of your settings.
Diagnostics > Factory Defaults > Factory Reset
After reboot, do not (like not at all) import your settings back in.
Make a minimal WAN LAN done setup. Ok the change the password - nothing more.
Now the system should be good. Time should be good. The resolver is resolving, not forwarding to 1.1.1.1 etc

@javen said in [sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100:

if the power is out longer than my UPS's battery can last.

When this happens, pfSense did a clean shut down by itself, right ?
Rebooting has to be done by removing the power shortly.

Have a look at the console interface while the system is booting, look for warning or error messages.

[javen]

I re-imaged the sg3100, installed to mmcds0 and then re-imaged to ada0 (the 32 GB M.2 drive I included in my sg3100 order).

I went through the 9-step wizard and still got the same behavior: DNS did not work after rebooting.
I didn't bother backing up my config or trying to restore a config--I want uncustomized, vanilla pfsense for now.

Disabling DNSSEC fixed the issue as before. I'm still getting teleported between Jan 20 2022 and Dec 27 2031.
Even when DNS is working, if I connect to the sg3100 via USB shell access, after rebooting:

[21.05.2-RELEASE][root@sg3100.localdomain]/root: date
Sat Dec 27 11:08:59 PST 2031

It seems like the system only gets the correct date when ntpd is able to reach the external ntp servers, which also requires having the dns interface working.

Of course I'm getting errors about bad certificates or keys during boot, because most keys have a validity period, with a validity window that probably doesn't extend to 2031. Shouldn't the system be able to keep its time from the last boot, and then correct any drift (hopefully by no more than a few seconds) the next time it fetches the current time from an ntp server? The coin cell battery in the sg3100 is +2.9V with the correct polarity. It doesn't make sense to me why the time resets every time the system reboots, and only is correct after the unbound has launched and the NTP service fetches the time from ntp.org. Every Linux system I have remember what time it is after rebooting before they connect to the network.

[keyser]

@javen Good diagnostics - so the time skew is definitively your real problem here.

Does your pfSense itself not get DNS servers from your WAN provider (via DHCP)?
You should make sure to setup your pfSense (itself) to access DNS either by default via your WAN DNS, or at least to fall back to those if unbound resolver goes offline.

You could fix your issue with using the override DNS feature on the System -> general settings page.

That way DNS will work for pfSense right away, and NTP will come up and time will be correctly set on the appliance.

EDIT: But I agree it's very weird that it does not keep time over a power cycle....

[Gertjan]

@javen said in [sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100:

Every Linux system I have remember what time

Except the ones that have a broken on board RTC ... ;)
Even iff the battery is still ok, change it.
Also : boot with the console - and goto the 'bios' : the clock will be chown. Maybe teher are some RTC settings ?? ( I don't own a 3100).
ntpd can be used, though : instead of using a host name like "fr.pool.ntp.org", you can use an IP.

@keyser said in [sg3100] [unbound] The DNS Resolver service must be manually rebooted every time I power cycle my sg3100:

Does your pfSense itself not get DNS servers from your WAN provider (via DHCP)?

If these still exist, they are meant to be used by the 'stupid' ISP routers that have nothing more as a simple forwarder.
pfSense is equipped with the far more superior "resolver".

I'm pretty sure there is no "unbound" issue what so ever, it's a 'time' issue.
I'm crossing my fingers for you, as 'mI hope the RTC device isn't broken.

[plfinch]

@javen Several of us are experiencing and have complained about this issue. Here is my thread on it. I gave up trying to fix it for now so still have to restart unbound after every reboot. Here is a pointer to my thread which may have some additional info for you…

Topic

Peter