having trouble with unbound not starting @ random after reboot

keyser

After upgrading my SG4200 to 26.07 I have issues with Unbound being very unreliable at startup. By random it does not startup after reboot and I have to login in to start it manually (which it does without problems).

From the log it’s clear at times that it tries to bind to interfaces that are not ready:
can't bind socket: Can't assign requested address for fe80::... port 53
fatal error: could not open ports

I have now tried to set it to listen on all interfaces instead of specific interfaces, and will be testing with that, but is this a known “racecondition” issue at startup?

Gertjan

@keyser

My LAN, using a igc0 NIC driver :

As you can see, the IPv6locallink uses fe80:: and the rest is created with the MAC address.
Same thing for the IPv6 GUA, if you use IPv6.

Do you use IPv6 ?

Your Status > Interfaces shows the fe80::.... ?

Your unbound version is the latest ? An update was available a couple of days ago.

Btw : I'm using a 4100 myself, and 26.07, and never bothered to select specific interfaces here :

as I need DNS on all interfaces.

edit :

I have now tried to set it to listen on all interfaces instead of specific interfaces,

I'm pretty sure that this would solve the issue.
But (asking to myself) : what is the real issue ??

@keyser said in having trouble with unbound not starting @ random after reboot:

can't bind socket: Can't assign requested address for fe80::... port 53
fatal error: could not open ports

Because it was already in use ?

Normally, interfaces are ready way before unbound, or any other process starts.
An local IPv6 local link IPv6 is also (imho) pretty static and won't change, even if the interface goes "down and comes back up".

keyser

@Gertjan There is no IPv6 setup in this instance. IPv6 is set to none on all interfaces, but it is not "disabled" using the advanced setting. So there is only the fe80:: link local addresses when it comes to IPv6 use.

I have not been able to test a lot yet, it's just very obvious an issue that has arisen and is a reboot/cold start issue, not actually a problem with Unbound not working/stable when I start it myself.

I'll follow up with results from setting the listening interfaces to "all"

patient0

@keyser it probably has nothing to do with your issue but have you updated to the lastest Unbound 1.26.1?

jimp mentioned it in another thread and how to update: https://forum.netgate.com/post/1249095

keyser

@patient0 But is that update not an "out of band" update - IE: outside the pfSense Release train, which could cause all kinds of unknown issues/things that does not survive a normal update on pfSense?

I'm rather reluctant to do that as I have (learned from experience) a reluctancy to do anything not supported and part of the procedure on my firewalls. I have had too many issues with things gone missing or failing to upgrade correctly after "tampering" outside the supported setup.

patient0

@keyser said in having trouble with unbound not starting @ random after reboot:

But is that update not an "out of band" update

Mmmh, I assumed if jimp writes about it that it either important enough to do (CVE's) and safe. But I have not had the experiences you had.

Maybe @jimp can clarify?

bmeeks

@keyser said in having trouble with unbound not starting @ random after reboot:

@patient0 But is that update not an "out of band" update - IE: outside the pfSense Release train, which could cause all kinds of unknown issues/things that does not survive a normal update on pfSense?

The Unbound package update should be very safe. Netgate pulled in the updated package source code into their package builder system for pfSense Plus and generated an updated package that is now available on the official pfSense repo. It's no different than installing an updated version of the System Patches package or any other officially supported package.

The only difference here with Unbound is that because it is normally stuffed into the larger "bundle" that is the pfSense install image, it is not listed as an individual package on the Available Packages menu. Instead, you do the update following the instructions posted by @jimp in https://forum.netgate.com/post/1249095.

The package update is to address a security CVE, so I doubt that it will directly impact the issue you are experiencing. That might be related instead to the issues of "startup race" that is being discussed in a few other threads.

keyser

@bmeeks Thank you for the very clarifying answer. I did not know this was actually something that negate did/was available in pfSense outside planned releases.

In this particular case I have no reason to suspect it's actually a problem with Unbound itself, so it likely won't affect the issue I'm having. But it's really good to know.

bmeeks

@keyser said in having trouble with unbound not starting @ random after reboot:

@bmeeks Thank you for the very clarifying answer. I did not know this was actually something that negate did/was available in pfSense outside planned releases.

Your initial caution is warranted. Never install a package from the public FreeBSD repos onto pfSense unless you want to live dangerously. There is always the risk of breaking the pfSense install due to differing shared library versions when installing a package from any non-pfSense (Netgate) repo.

In this case with Unbound (and in the past with a few other binary packages), Netgate fetched the updated code from upstream and rebuilt a pfSense-compatible package in their build environment then posted it on the official pfSense package repo. Expect this scenario to become more common as pfSense moves from a large single-image install process over to the new FreeBSD pkg base process where the install is split out into the individual packages. This makes future incremental updates easier since you can update just a single package. Again, though, even this pkg base process requires you to install packages only from the official pfSense repo.

SteveITS

@keyser also anyone upgrading from an older pfSense version to current since that was updated will receive the new version. (Edit: they just slipstreamed into the current branches…early upgraders unfortunately need the manual step.)

I’d expect binding unbound to all will help, let us know. You can control inbound access by firewall rules.

Gertjan

@patient0

He did Unbound 1.26 available and because he (they/Netgate) waited a it bit before releasing (building) 1.26 for pfSense. that why we have the now newer 1.26.1, with includes even more "CVE's" (and probably some bug fixes on the bug fixes that '1.26' introduced ^^^

@keyser said in having trouble with unbound not starting @ random after reboot:

But is that update not an "out of band" update (?)

Maybe.
But Netgate has released it, so it will be part of the future pfSense.

When you get a notification like (example) :

13:15:11 Some packages are part of the base system and will not show up in Package Manager. If any such updates are listed below, run `pkg upgrade` from the shell to install them:

......
unbound: 1.25.2 -> 1.26.1 [pfSense]

then you know Netgate uses it, and so maybe you can use it also.
These upgrades are not some sort of 'test drive' upgrade.

If, for example, unbound 1.26.1 becomes mandatory, because '1.25.2' has some known major issue, then pfSense itself would be upgraded, like : from 26.07-RELEASE to 26.07.1.

Btw : The the unbound 1.26.1 update is just the package file itself and probably some other unbound package related files. There's no pfSense GUI upgrade part, as no new or modified options are exposed in the GUI.

Gertjan

@patient0

He did, see here : Unbound 1.26 available and because he (they/Netgate) waited a bit before releasing (building) 1.26 for pfSense. That why we have the now newer 1.26.1 version, with includes a big "CVE's" list and probably some bug fixes on the bug fixes that '1.26' introduced ^^^

@keyser said in having trouble with unbound not starting @ random after reboot:

But is that update not an "out of band" update (?)

Maybe.
But Netgate has released it, so it will be part of the future pfSense.

When you get a notification like (example) :

13:15:11 Some packages are part of the base system and will not show up in Package Manager. If any such updates are listed below, run `pkg upgrade` from the shell to install them:

......
unbound: 1.25.2 -> 1.26.1 [pfSense]

then you know Netgate uses it, and so maybe you can use it also.
These upgrades are not some sort of 'test drive' upgrade.

If, for example, unbound 1.26.1 becomes mandatory (Netgate's decision), because '1.25.2' has some known major issue, then pfSense itself would be upgraded, like : from 26.07-RELEASE to 26.07.1.

The the unbound 1.26.1 update is just the package file itself and probably some other unbound package related files. There's no pfSense GUI upgrade part, as no new or modified options are exposed in the GUI.

edit : my very 'imho' of course.