Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0

jsokorp

I am seeing what appears to be the same issue as Redmine #13707 on pfSense CE 2.9.0-RELEASE.

This has happened repeatedly after power outages or cold boots. My town is having repeated power outages so this has become quite annoying as I was locked out of the web interface each time and just discovered the workaround.

DNS Resolver shows as running, but LAN clients cannot resolve anything. The pfSense GUI hostname also fails because LAN DNS is unavailable.

During the most recent failure:

sockstat -4 -l | grep ‘:53’
unbound unbound 67000 3 udp4 127.0.0.1:53 :
unbound unbound 67000 4 tcp4 127.0.0.1:53 :

/var/unbound/unbound.conf contained:

interface: 127.0.0.1
interface: ::1

However, DNS Resolver > Network Interfaces was explicitly configured for:

LAN (192.168.10.1)
OPT1 (192.168.50.1)
Localhost

Unbound itself appeared healthy:

unbound-control -c /var/unbound/unbound.conf status
version: 1.25.2
threads: 4
modules: 2 [ validator iterator ]
unbound is running

There were no obvious Unbound startup errors in /var/log/resolver.log.

I then went to Services > DNS Resolver and clicked Save / Apply without changing the interface selections.

Immediately afterward /var/unbound/unbound.conf was regenerated as:

interface: 192.168.10.1
interface: 192.168.50.1
interface: 127.0.0.1
interface: ::1

sockstat then showed Unbound listening on all of those IPv4 addresses, and LAN DNS immediately worked again.

It looks like the Unbound configuration may be getting generated during boot before the selected interfaces are available. Saving the same configuration later generates the expected interface entries.

I have now changed Network Interfaces to All as a workaround, as discussed in Redmine #13707.

Relevant configuration:

pfSense CE 2.9.0-RELEASE
Unbound 1.25.2
Resolver mode, not forwarding mode
DNSSEC enabled
Outgoing Network Interfaces: All
Access Lists: empty
Auto-added access control enabled

Is #13707 still applicable to 2.9.0?

If this happens again, is there any specific boot-time diagnostic information I should collect before touching the DNS Resolver configuration?

DBMandrake

@jsokorp And four years after creating that redmine ticket (I forgot about that ticket!) I still rely on the workaround of binding unbound to all interfaces and controlling access on different network interfaces via firewall rules... (which is actually a reasonable way to do it, from a practical perspective)

There are shortcomings in the way interfaces are brought up and services started/restarted that lead to this problem (I did dig into it back when I first reported the problem, but don't remember the fine details anymore) and I suspect it's quite a lot of work to refactor all that stuff to truly fix this problem.

I tried to help diagnose the root cause further but Jim just wanted to shut me down and shunt me back to the forum (where he would not engage) claiming that the very obvious race conditions in the startup scripts that I had identified don't exist and that my situation was "just an edge case", so at that point I gave up trying to help people who don't want help, as my time is valuable too and I had a satisfactory workaround.

It's good to remind Netgate this problem still exists but from a practical perspective my advice is to simply bind it to all interfaces and control access via firewall rules and move on! This is unlikely to be fixed properly any time soon, given the redmine ticket is now four years old with no resolution or even an acknowledgement that this is a real problem.

luckman212

@DBMandrake just posted this over on the redmine #13707, but… copying it here too for completeness:

I haven't looked deeply into this bug, but since check_reload_status appears frequently in the log messages, I assume the race condition may be helped by the fixes going on at https://github.com/luckman212/check-reload-status. Anyone experiencing this who has the time and inclination, may want to install that new version and re-test.

jsokorp

@DBMandrake Thanks for the reply. I appreciate it.

jsokorp

@luckman212 said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

https://github.com/luckman212/check-reload-status

I took a peak at "check-reload-status" and I am interested in testing but first want to be sure Layer 8 is not your mantra.

And you suggested I was using an LLM for my post? —

jsokorp

@luckman212

I built your patched daemon and companion utilities from source on an amd64 FreeBSD 16 VM and manually installed them, including the gateway-reconciliation helper, on pfSense CE 2.9.0. I enabled DEBUG logging and rebooted.

The daemon remained running. The logged FastCGI requests for gateway reconciliation, filter synchronization, and a later interface link-up all completed. I did not see failures for those requests.

However, it did not resolve my Unbound startup issue: my saved DNS Resolver interface selections remain configured, but the LAN UDP/TCP listeners are missing after startup. Selecting “All” has previously avoided this.

I have a separate script that waits for pfSense’s boot marker to clear, checks the configured listener addresses, and calls services_unbound_configure(false) if listeners are missing. This reboot showed:

16:12:37 guard: Expected UDP/TCP listeners missing; regenerating Unbound configuration once
16:12:38 guard: Reconfiguration finished: exit=0 elapsed=1s
16:12:43 guard: Unbound UDP/TCP listeners restored

I subsequently confirmed both LAN listeners with sockstat and a successful DNS query. Thus, recovery followed the script’s intervention; the patched daemon had not prevented the missing-listener condition.

The boot log also contains “Ignoring link event during boot sequence.” The installed rc.linkup deliberately returns at that point while the boot flag is set, with a limited DHCP exception. The log does not identify that event’s interface, so I cannot connect it to the Unbound issue.

Gertjan

@jsokorp

Before every unbound start, the unbound.conf file is regenerated.

A list with interfaces selected, by the admin, is retrieved from here.

This could be 'all' :

or a list with specific interfaces.

Then, for every admin/GUI selected interface in the unbound config, it's tested if it's 'up' and thus available / know and active for the system at that very moment).

The remaining list with valid interface is added to the unbound config file here.

So, when the list is selected interface is parsed, and at that very moment the system (the OS !) informs the script that the (a) interface is unavailable at that very moment, the 'interface: x.x.x.x' line isn't added and unbound will start without dns support on that interface.
This is a good thing, as unbound can't be started with interfaces that do not exist, it will fail right away.
Interfaces can go down and up at any time. They really should not be doing that, but have a look at the forum posts : many pfSense admins have issues when interfaces are flapping all the time.
After all, interfaces can go bad (both sides), the cable can be bad, or the switch power brick has a poor quality.
Connecting an end user device like a PC directly to a pfSense LAN NIC is also not a good idea (imho).

jsokorp

@Gertjan

Thanks—that fits what I’m seeing. The selected interfaces are available after boot, and regenerating Unbound’s configuration then restores the listeners. I suspect a startup-order or readiness issue elsewhere is contributing. The remaining question is what causes configuration generation to miss those interfaces during boot.

Gertjan

@jsokorp

The unbound start (upon boot) isn't the only process that starts at that moment.
For example, dhcp, nginx, php and many other get also started at nearly the same moment.
These processes are just listening, and do anything with interfaces.*
But other processes, like a VPN client (more exist !), does something with interfaces.
So, when the a VPN client starts, a new interface is created, and the system gets a global notification that 'an interface event' happened.
The question is : what pfSense packages (and other pfSense options) are you using ?
More precise : is there a potential candidate that 'resets' the interfaces when it starts ?
As processes are started rapidly one after the other, and unbound was the process just starts moment later and it found the physical interfaces down (for a short moment) and there you have the situation explained.
Question in short : what starts ? (what do you use ?)

jsokorp

@Gertjan

My setup* may influence the timing, but I identified a specific interaction in pfSense’s startup code that explains the persistent failure.

The default Network Interfaces selection is All, which takes a different code path: pfSense generates interface-automatic: yes rather than building an explicit list through the per-interface status checks below. That distinction could explain why users retaining the default do not encounter this particular failure.

With explicitly selected interfaces, /etc/inc/unbound.inc skips interfaces whose status is not up. Temporary diagnostics confirmed that LAN was enabled but reported no carrier when that configuration was generated. LAN then came up during boot, but /etc/rc.linkup logged that its link events were being ignored because boot was still in progress. The LAN listener remained missing until Unbound’s configuration was regenerated.

As a test, I added a carrier-aware wait before generation. It waited 2.63 seconds, and Unbound started with its LAN listener without my recovery script intervening. The original failure also reproduced with two different switch models, negotiating at 1 Gb/s and 2.5 Gb/s.

So I agree there may be something about my setup that exposes the timing window. But the software issue appears to be that an interface omitted during initial configuration becomes available during boot, and the skipped configuration is not subsequently reconciled. Is there an existing boot-completion mechanism that is supposed to handle that?

*My setup, to answer your question about what is installed and starts:

pfSense CE 2.9.0 on Protectli hardware with Intel igc interfaces. WAN uses DHCP; LAN has a static IPv4 address. IPv6 configuration is set to None on WAN and LAN.
LAN normally connects to a TP-Link TL-SX105 unmanaged switch, which connects to an EnGenius EWS2910P-FIT managed switch. The failure also reproduced with pfSense connected directly to the EnGenius, and separately with the EnGenius disconnected.
Unbound listens on explicitly selected LAN and localhost interfaces. Its outgoing interface is an assigned PIA OpenVPN client interface, with Quad9 DNS-over-TLS upstreams.
Kea DHCP is enabled on LAN.
Installed packages: pfBlockerNG, HAProxy, FreeRADIUS, Suricata, Tailscale, WireGuard, ACME, net-snmp, Shellcmd, System Patches, Notes, iperf, and nmap. This is the installed inventory; it does not mean every service is enabled.
Tailscale and WireGuard are disabled, and no WireGuard tunnels are configured. The problem existed before Suricata was installed.

Shellcmd has four startup entries:

Early: the package-created WireGuard service command, which reports that WireGuard is disabled.
Early: the System Patches apply command.
Normal: our custom PIA DNS/VPN recovery watchdog.
Normal: our custom Unbound listener recovery guard.

The two custom recovery scripts were added to address the problem; it predates them.

I have not identified a service that resets LAN. In the diagnostic boot, the LAN UP/DOWN messages appeared during “Configuring LAN interface,” before “Syncing OpenVPN settings.” That seems relevant to where I should look for the initial transition, although it does not identify the exact operation responsible.

Gertjan

@jsokorp said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

The default Network Interfaces selection is All, which takes a different code path: pfSense generates interface-automatic: yes rather than building an explicit list through the per-interface status checks below. That distinction could explain why users retaining the default do not encounter this particular failure.

Wow ....
Thanks !! This explains very well why 'all' doesn't (can't !!) complain about missing interfaces.

so unbound will bind (== listen) on all interfaces it finds when it starts.
If some interface is missing then that's no issue. unbound will find all local hosts interfaces, like 127.0.0.1 and ::1, and every other interface that happens to be up at that moment.

The thing is, you've found a :

unbound unbound 67000 3 udp4 127.0.0.1:53 :
unbound unbound 67000 4 tcp4 127.0.0.1:53 :

Did the missing interfaces came up ? == where active and present ?
Normally, and interface UP event would trigger a new "stop unbound, generate a new config, and start unbound' sequence.
During this new sequence, an LAN or WAN type interface would have been found, and if selected, included into the 'unbound listen' list.
Or, clear enough, this didn't happen.
There must be some small time windows where interface events are discard... ?
I guess, to prevent race conditions.

DBMandrake

@Gertjan

@Gertjan said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

so unbound will bind (== listen) on all interfaces it finds when it starts.
If some interface is missing then that's no issue. unbound will find all local hosts interfaces, like 127.0.0.1 and ::1, and every other interface that happens to be up at that moment.

It's more than that. In FreeBSD/Linux etc when you bind a daemon to "all" interfaces (binding to 0.0.0.0, which is a special case) it means all currently active interfaces AND all future interfaces that come up later while the daemon is already running. This is why it can never have a problem with the "all" setting.

As soon as you limit binding to a list of specific interfaces, you can ONLY bind to interfaces that are actually up right now.

That's why the scripts look at the list of interfaces you have requested in the UI but only bind to those that are actually up at the time the script runs. If you specify OPT2 as an interface to bind to but it is not up yet, you can't bind to it right now, so the configuration generated for unbound excludes it even though you asked for it.

It then becomes the responsibility of a script to re-configure and re-launch unbound if an interface that was previously wanted but was not yet available, becomes available. (And also the reverse)

There is at least one race condition between the bootup scripts and the automatic interface up/down scripts that is triggered during boot if the timing of your network interface coming up is unlucky and that is the root cause of this issue.

But fixing it properly is not necessarily easy or obvious and probably requires a bit of rethinking of the whole way this is done in PFSense, which is currently a bit brittle.

Gertjan

@DBMandrake said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

It then becomes the responsibility of a script ...

Afaik, when an interface events happens, like (your example) OPT2 comes back online (active), then this system event will wind up restarting process, like unbound. That is : it will be stopped, the unbound.conf is rebuild, and this time OPT2 is selected, and this time its found to be online (active) so it will be included.

I agree, there is a moment when this uplink event(s) was (are) missed, so no subsequent UP link events are triggered, and an interface (your case is : all local physical interfaces !!) are skipped.

jsokorp

An update: I now have a GUI-configured workaround that restored the LAN DNS listener on the first reboot tested.

Previously, with LAN explicitly selected under DNS Resolver → Network Interfaces, temporary diagnostics showed that LAN was enabled but reported no carrier when Unbound’s configuration was generated. The selected-interface code in /etc/inc/unbound.inc skipped it. LAN subsequently came up, but its link events were ignored while boot was still in progress, and the missing listener was not restored until Unbound’s configuration was manually saved and applied.

The change was:

Set Network Interfaces to Localhost only.
Add these directives under the existing server: section in Custom Options:

interface: 192.168.10.1
ip-transparent: yes

This leaves the LAN interface itself configured normally. It explicitly supplies Unbound’s static LAN listening address instead of relying on the interface-selection code to include it during boot.

After Save/Apply and a reboot, the relevant log sequence was:

17:22:39 Starting DNS Resolver...done.
17:22:39 igc1: link state changed to UP
17:22:41 Bootup complete

The LAN listener was present on both protocols:

# sockstat -4 -l | rg 'unbound.*192\.168\.10\.1:53'
unbound unbound 62335 7 udp4 192.168.10.1:53 *:*
unbound unbound 62335 9 tcp4 192.168.10.1:53 *:*

A query sent directly to it succeeded:

# drill @192.168.10.1 example.com A
rcode: NOERROR
ANSWER: 2
SERVER: 192.168.10.1

My listener guard reported action=none, and there were no guard-reconfiguration messages in the boot excerpt.

This supports the diagnosis that omitting an explicitly selected interface during a temporary no-carrier state, without subsequently regenerating the configuration, is the mechanism behind the missing listener. It does not establish why carrier became available at that particular point during boot.

Is this explicit-address configuration an appropriate workaround for a static LAN address, and should pfSense revisit how selected Unbound interfaces are handled when carrier is temporarily unavailable during boot?

This was tested on pfSense CE 2.9.0 with unofficial modified versions of check_reload_status, pfSctl, and fcgicli installed. The workaround has been verified on ~~one~~ two reboots so far.

SteveITS

If it is just an issue at boot, possibly another workaround could be to use the cron package to run a short script @reboot to wait a few seconds then restart unbound.

keyser

@Gertjan Just thinking out loud here, but I am experiencing the exact same problem on systems where I use Tailscale. I see OP also uses Tailscale, so perhaps this is what’s causing the interface event at bootup?

Gertjan

@keyser said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

... thinking ....

Not a solution, but you could stop 'thinking' (not being sure) about it, as you could be 'sure' about it
If possible, stop using tailscale for a while.
If it was tailscale that 'did something' with the interfaces, then that issue stops.
The issue continues ? Then tailsscale is excluded.

keyser

@Gertjan I can't in my setup as tail scale is in "production" on that setup, but perhaps the OP can test that.

jsokorp

@keyser Thanks—that’s a useful comparison. Tailscale is installed on my firewall, but it is disabled, and the Unbound boot issue still occurs.

My boot diagnostics captured Unbound generating its configuration while LAN was enabled but reported “no carrier.” LAN subsequently came up, but its link event was ignored during the boot sequence, leaving Unbound without its LAN listener.

So this reproduces with Tailscale disabled and never configured. I haven’t tested with the package completely uninstalled, so I can’t yet rule out its startup integration, but an active Tailscale connection isn’t required for the failure on my system

Also I did post above that I found a solution for my setup using the GUI. Maybe give that approach a shot and see if it works for you.

For me the issue was merely having LAN selected in Network Interfaces in ServicesDNS/ ResolverGeneral/ Settings.

Gertjan

@jsokorp said in Unbound binds only to localhost after cold boot/power outage on pfSense CE 2.9.0:

So this reproduces with Tailscale disabled and never configured. I haven’t tested with the package completely uninstalled, so I can’t yet rule out its startup integration, but an active Tailscale connection isn’t required for the failure on my system

I tend to think that the issue isn't 'tailscale' based then.
The package is installed, and only occupies disk space. pfSense isn't using it.