Unbound DNS Resolver not starting

[KKIT]

My DNS Resolver stopped working a few days ago and now I just saw that the service can not be started.

Setup:
2x Netgate 7100 in HA
pfSense Plus 24.03

PKGs installed:
System-Patches with recommended patches installed
pfBlockerNG, though I removed it now without retaining the settings and rebooted
OpenVPN Export

The resolver is working on the backup node which makes me think that pfBlocker did something to the unbound config?

I tried:
Reboot
Change a setting in the advanced section, then restart the service
Confirm my settings with backup node but last changes I made are months ago now.

Help is greatly appreciated, as I need the node back up and running.

Logs:
https://pastebin.com/irBf12mY

[Gertjan]

@KKIT

The logs show the ususal stop and start sequence.
When it stops, it dumps several line of statistics.
The log - up until line 75 - is dumping starts.
Then, at line 76, 3 seconds later, unbound is started again ...
Line 78 : its told (signaled) to stop again ( ! )

Line 95 (3 seconds later) there is another stop and start sequence.

Next time, line 109, it holds for nearly two days.

---

Just to be sure : this one is unchecked, right ?

login-to-view

( I guess it it).

Other reason why a process like unbound gets restarted : an interface is used (bound to) is taken down for a moment. That will restart attached processes, and again when it comes back.
Some one is ripping out cables ?

[KKIT]

@Gertjan

I disabled a few interfaces a few days ago but that change also reflected on the backup node. Yes, DHCP Registration is disabled. WHat I don't understand is why the logs are not showing my startup attempts.

Question: Is it possible to reinstall unbound? Or maybe I reset the config? If so, how would I go about that?

[KKIT]

@Gertjan Also, do you know if pfBlocker could have messed with unbound?

[Gertjan]

@KKIT said in Unbound DNS Resolver not starting:

Question: Is it possible to reinstall unbound?

Noop. I don't think so.
It's just an executable. And some support files like unbound-control etc.

@KKIT said in Unbound DNS Resolver not starting:

Or maybe I reset the config?

This is pfSense.
This is FreeBSD.
This means : we all have the same identical binary executable Library etc files.
Only our "config file" file is different : it's here /var/unbound/unbound.conf
and this file is re generated just before pfSense starts unbound.
Like any other process on the system.

@KKIT said in Unbound DNS Resolver not starting:

If so, how would I go about that?

I would : open a console or SSH to pfsense, go médecin mode (option 8) and :

tail -f /var/log/resolver.log

Now, in the GUI, interact with unbound and see what the log tailing tells you.
In another SSH session, medecin mode, use :

top

On another SSH, option also, use

dig @127.0.0.1 google.com +trace

to question unbound with a hots name lookup. You should repeat this test as often as possible.
Normally, unbound should listen to 127.0.0.1 - and you should receive an answer.

Ask if unbound is running, lists on what interface it listens :

24.03-RELEASE][root@pfSense.bhf.tld]/root: sockstat | grep 'unbound'
unbound  unbound    39327 3   udp6   *:53                  *:*
unbound  unbound    39327 4   tcp6   *:53                  *:*
unbound  unbound    39327 5   udp4   *:53                  *:*
unbound  unbound    39327 6   tcp4   *:53                  *:*
unbound  unbound    39327 7   tcp4   127.0.0.1:953         *:*
unbound  unbound    39327 8   dgram  -> /var/run/log
unbound  unbound    39327 10  stream -> [39327 12]
unbound  unbound    39327 12  stream -> [39327 10]
unbound  unbound    39327 13  stream -> [39327 14]
unbound  unbound    39327 14  stream -> [39327 13]

The first 4 lines means : unbound listens on port 53 :
For UDP and TCP
For IPv4 and IPv6
For every interface

Btw : these are not really 'pfSense instructions'. Not even FreeBSD instruction - or Linux for that matter. These commands are - with some nuances - avaible on every OS on planet earth ^^

[KKIT]

@Gertjan said in Unbound DNS Resolver not starting:

tail -f /var/log/resolver/log

Thanks for your suggestions, right off the bat, I get this:
tail: /var/log/resolver/log: No such file or directory

I went in with option 8, as you mentioned. So I guess there is a deeper rooted issue. Do you suggest I reinstall pfSense and load from a backup?

[patient0]

@KKIT I did fiddle with the Unbound settings yesterday and wasn't able to start Unbound. In my case it turned out that Unbound wasn't stopped cleanly and the therefore couldn't bind to port 53.

Maybe check that no Unbound instance is running. Under Diagnostics / Command Prompt and run:

ps aux | fgrep -i unbound
unbound 38035   0.0  4.3 210648 175800  -  Is   16:11      0:47.64 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root    52767   0.0  0.1  13400   3020  -  S    14:20      0:00.00 sh -c ps aux | fgrep -i unbound 2>&1
root    53148   0.0  0.1  12832   2496  -  S    14:20      0:00.00 fgrep -i unbound

And set LogLevel to at least 2 for the testing.

[fireodo]

@KKIT said in Unbound DNS Resolver not starting:

tail: /var/log/resolver/log: No such file or directory

tail -f /var/log/resolver.log

[KKIT]

@patient0 said in Unbound DNS Resolver not starting:

unbound 38035 0.0 4.3 210648 175800 - Is 16:11 0:47.64 /usr/local/sbin/unbound -c /var/unbound/unbound.conf

I get this when putting in line 2

login-to-view

[patient0]

@KKIT I'm sorry I wasn't clear: you enter only the first line ps aux | fgrep -i unbound and the other lines are the result of that command in my case.

[Gertjan]

@KKIT

Nooooo.
Never - ever use the GUI Diagnostics > Command Prompt
GUI command line is ... bad, or worse.
Use SSH.

[KKIT]

@fireodo said in Unbound DNS Resolver not starting:

@KKIT said in Unbound DNS Resolver not starting:

tail: /var/log/resolver/log: No such file or directory

tail -f /var/log/resolver.log

I did and nothing changes in the log when I try to adjust anything
login-to-view

[KKIT]

@Gertjan

Thanks, I get this:

root 64053 0.0 0.0 13412 3064 - S 14:46 0:00.00 sh -c ps aux | fgrep -i unbound 2>&1
root 64286 0.0 0.0 12840 2532 - S 14:46 0:00.00 fgrep -i unbound

[Gertjan]

@KKIT said in Unbound DNS Resolver not starting:

I did and nothing changes in the log when I try to adjust anything

With :

login-to-view

you will only see the start and stop lines.
When a stop arrives, unbound will log about 30 "info" lines, the ones you've showed above.

Btw : when you change edit whatever do something with the 3 unbound (revolver) settings pages, unbound won't get restarted. So no log lines.
Only when you click on Apply on the top of any resolver settings page, unbound will get stopped, a new config file gets created, and unbound gets started. Then you see the the stop announcement, dump of the info and the the restart line.

With a log level of 3, 4 or a whopping 5 you will see much more.
Remember to reset to "1" when you're done (and apply ^^), as with level 5 unbound will log huge quantities.

[patient0]

@KKIT said in Unbound DNS Resolver not starting:

@Gertjan

Thanks, I get this:

root 64053 0.0 0.0 13412 3064 - S 14:46 0:00.00 sh -c ps aux | fgrep -i unbound 2>&1
root 64286 0.0 0.0 12840 2532 - S 14:46 0:00.00 fgrep -i unbound

Ok, that means no Unbound instance is running at that time. Next I'd set the LogLevel to > 1.

Do you have a fairly standard Unbound config? Running as a resolver or forwarder? Nothing else listening to port 53?

[KKIT]

@Gertjan

Unfortunately absolutely nothing happens when I set it to 5, I am still connected via SSH and within the logfile with tail, I also made sure to apply the settings as well as manually start the service under the "Services" section, nothing :(

[KKIT]

@patient0

I set the log level to 5 and still nothing, only the second node is listening on port 53 and the resolver is running as expected. No custom configuration on unbound and no DNS forwarder active.

[Gertjan]

@KKIT

I've just :

tail -f /var/log/resolver.log

and then :

login-to-view

and then

login-to-view

and oh boy .... the tail command had a hard time keeping up with the pace. tens of lines a second, impossible to follow.

Btw : I've no HA setup, just one box, and about 5 PC connected and some other stuff.
I know that every device in my network uses my 'pfSense' as a DNS source, not some 1.1.1.1 or 8.8.8.8 or other data collector.

If your unbound doesn't log anything on level 5 : isn't that a sign that LAN devices uses another device, and not (this !) pfSense, to do DNS requests ?

Another trick : GUI this time ^^

login-to-view

Set interface to your LAN interface, or any other LAN type interface.
Set the port to '53' (DNS, recall)
And hit Start.

Now you should see "destination port 53".

Add the IP of your pfSense, so now you'll see what LAN devices want to talk to your pfSense for DNS needs :

login-to-view

Btw : I presume the default 192.168.1.1
if there is no ore little DNS traffic, its normal that unbound doesn't 'log' ^^

[KKIT]

@Gertjan
I appreciate the input, but I think we are looking at the wrong end here. The Unbound service is not running at all, as confirmed above. Additionally, the node is currently in maintenance mode, I already tried putting it back into production but I wouldn't accept any requests. At this point I assume that either pfBlocker messed it up or I improperly shut the service down. I will go for a reinstall and see from there. THanks so far, really appreciate it and will keep you guys updated!

[KKIT]

@KKIT

So I wanted to give a followup on this issue, I dug a little deeper and looking at this post:
https://forum.netgate.com/topic/154372/unbound-dns-resolver-will-not-start/3

I am pretty sure it has something to do with pfBLockerNG messing up my config file. I checked the directory of the config file and see that there are two with one having a ".error" added to it:

login-to-view

Trying to reload unbound via shell shows this error:
unbound-control[85585:0] error: connect: Connection refused for 127.0.0.1 port 953

Unfortunately I lost my backups after the reinstall so my question is if I can transfer the unbound.conf from my functioning pfSense (Node B) to make it work again?