Unbound resolver error: Can't assign requested address for 127.0.0.1

themadsalvi · Mar 26, 2019, 2:29 PM

Hello everyone,
It appears as though an unbound resolver issue has hit my pfsense. I have tried pretty much everything that is simple. I even went as far as doing a factory reset, and setting everything up from a working configuration from last year. So far nothing has fixed the issue of this error popping up when I force reload DNSBL from pfblocker. The reason for the force reload is that the DNSBL is always out of sync, and sites will not load/resolve correctly. I have read another thread about this, and the user found that unbound_control.key and unbound_server.pem had been corrupted from a power outage. He was able to get them recreated, but did not post how. We had a power outage recently in my neighborhood right before this problem occurred.

I wanted to know if there was a way to get pfsense to recreate these two files/keys in the GUI(getting to the console from there, etc)? I believe that @BBcan177 helped in this situation. Thank you for your time

Gertjan · Mar 26, 2019, 4:33 PM

Hi,

Please confirm that you are using 2.4.4-p2.

I advise you to isolate the problem.
First : de activate packages like pfblocker.
In the pfSEnse GUI : goto Status => Services and stop the Resolver.
On another browser windows, visit Status > System > Logs > System > DNS Resolver
In the first browser window, start the Resolver.
In the second windows, hit Refesh (probably F5).

Show us the log (reversed order) :

Mar 26 15:46:59 	unbound 	12158:0 	info: start of service (unbound 1.8.1).
....
Mar 26 15:46:30 	unbound 	13394:0 	info: service stopped (unbound 1.8.1).

About the

The manual states :

# create certificate keys; do not recreate if they already exist.

So, delete these two files (2 x .key file / 2 x .pem file) and restart unbound.

Btw : this looks like a managed file system error. Certs are there, but can't be recreated - and contain wrong info.
It's probably time to use the console access **, and execute a couple of times fsck.

Power outage and the resulting hard shut down of pfSense ? Never saw that before.
pfSense behaves like a Windows PC these days : pull out the plug and you have a good chance it won't wake up again.
That's why UPS exist ... use them to remove a lot of possible problems.

** correction : if you do not use an UPS you should be an console access expert. That's one more reason why people prefer using an UPS ^^

themadsalvi · Mar 26, 2019, 4:42 PM

@Gertjan
these are the logs that I am able to get from the DNS resolver!login-to-view login-to-view

I will get a better UPS, as I have a 600VA one at the moment. It was not enough for this situation

Edit: Placed the DNSBL reload error below for good measure

login-to-view

I am in the shell, but unsure of commands to use to delete the files along with recreating them.

Gertjan · Mar 26, 2019, 4:46 PM

unbound listens to port 953 :

[2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: sockstat -4l | grep 'unbound'
unbound  unbound    28162 6  udp4   127.0.0.1:53          *:*
unbound  unbound    28162 7  tcp4   127.0.0.1:53          *:*
unbound  unbound    56848 6  udp4   *:53                  *:*
unbound  unbound    56848 7  tcp4   *:53                  *:*
unbound  unbound    56848 8  tcp4   127.0.0.1:953         *:*

This :
login-to-view

means probably that DNSBL tries to restart unbound, but it (re) started it to fast - unbound wasn't stopped (etc) and thus port '953' remains 'occupied'.
The new unbound instance can't grab it - and complains about it after stopping.

Run

[2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ps ax | grep 'unbound'
11558  -  Ss       0:00.22 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
48881  1  S+       0:00.00 grep unbound

to see what happens on your pfSEnse.
If needed, stop unbound using the GUI, and if any zombies left, kill them.
Using the kill command and the process number.

edit : By any chance : your are not trying to overload unbound == very long startup time (by importing a huge number of DNSBL). On very small systems big lists can make unbound very slow to start, stop and just operate.

themadsalvi · Mar 26, 2019, 5:00 PM

@Gertjan said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

unbound listens to port 953 :
[2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: sockstat -4l | grep 'unbound'
unbound  unbound    28162 6  udp4   127.0.0.1:53          *:*
unbound  unbound    28162 7  tcp4   127.0.0.1:53          *:*
unbound  unbound    56848 6  udp4   *:53                  *:*
unbound  unbound    56848 7  tcp4   *:53                  *:*
unbound  unbound    56848 8  tcp4   127.0.0.1:953         *:*
This :
login-to-view

means probably that DNSBL tries to restart unbound, but it (re) started it to fast - unbound wasn't stopped (etc) and thus port '953' remains 'occupied'.
The new unbound instance can't grab it - and complains about it after stopping.

Run
[2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ps ax | grep 'unbound'
11558  -  Ss       0:00.22 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
48881  1  S+       0:00.00 grep unbound
to see what happens on your pfSEnse.
If needed, stop unbound using the GUI, and if any zombies left, kill them.
Using the kill command and the process number.

edit : By any chance : your are not trying to overload unbound == very long startup time (by importing a huge number of DNSBL). On very small systems big lists can make unbound very slow to start, stop and just operate.

This is before I disabled the resolver
login-to-view

After I disabled it, the grep command came up with this
login-to-view

Then I killed the remaining 79387 process. The other process came up with a "no such process". Did I do this right? This comes up when doing the grep command after restarting the unbound resolver

login-to-view

RonpfS · Mar 26, 2019, 5:01 PM

@themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

DNSBL is always out of sync,

Can you post the pfblockerng log during a Force Reload DNSBL so we can see why you get the Out of Sync errors ?

You can get those errors when you have duplicate Headers / Label in DNSBL.

How much memory do you have on that system ? 8GB can support around 1000000 DNSBL entries.

themadsalvi · Mar 26, 2019, 5:17 PM

@Gertjan @RonpfS said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

@themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

DNSBL is always out of sync,

Can you post the pfblockerng log during a Force Reload DNSBL so we can see why you get the Out of Sync errors ?

You can get those errors when you have duplicate Headers / Label in DNSBL.

How much memory do you have on that system ? 8GB can support around 1000000 DNSBL entries.

I currently have 4GB, which is 45% percent used according to pfsense. I can add more, but I have been running it since last year with no issues.

Below is a raw dump of the pfblockerng log in a text file(too many characters to do a full dump:

pfblockerng.txt

RonpfS · Mar 26, 2019, 5:27 PM

Those tables : pfB_PRI1_v4, pfB_PRI4_v4, pfB_PRI2_v4, DNSBL_pfB_PRI2_v4 - pfB_PRI2_v4, DNSBL_Abuse - pfB_Abuse_PS_v4 shouldn't be in DNSBL, they are IPv4 tables, remove them.

Disable BBC_DGA it's probably too big for your memory. And try another Force Reload DNSBL.

You have to monitor memory usage with Status Monitoring. The Dashboard only display "current" memory usage, the Monitoring will give you memory usage over time.

themadsalvi · Mar 26, 2019, 5:43 PM

@RonpfS

Removed those, and forced a reload, which still had the unbound resolver error.

This is the result in the status monitoring during and after reload
login-to-view

This is the force reload log
pfblockerng2.txt

RonpfS · Mar 26, 2019, 5:58 PM

You still have pfB_Abuse_PS_v4 to remove
Try again with BBC_DGA feed disabled.
If it still fails, then post your DNS Resolver config.

themadsalvi · Mar 26, 2019, 6:11 PM

@RonpfS @Gertjan
Here is the latest file for the reload, with all of the lists gone that you told me to delete. Same error pops up:
pfblockerng3.txt

login-to-view

Rsolver settings.

login-to-view

RonpfS · Mar 26, 2019, 6:22 PM

Did you try to remove the private-domain: line ?
On my box I have Prefetch Support and Prefetch DNS Key Support ticked.

themadsalvi · Mar 26, 2019, 6:29 PM

@RonpfS @Gertjan
I ended up taking the private domain line out(save and apply), then checking the prefetch support and Prefetch DNS Key Support boxes(save and apply changes). Tried the forced reload, with those changes, and the error persists.

RonpfS · Mar 26, 2019, 6:31 PM

In a shell or Diagnostics Command prompt, do a

ls -al /var/unbound /var/db/pfblockerng

themadsalvi · Mar 26, 2019, 6:41 PM

@RonpfS @Gertjan
I have placed the output below

login-to-view
login-to-view

Why are the last 4 so old?

RonpfS · Mar 26, 2019, 6:52 PM

@themadsalvi The 2012 timestamp looks suspicious compared to mine :

-rw-r-----   1 unbound  unbound       2459 Dec  8 19:42 unbound_control.key
-rw-r-----   1 unbound  unbound       1330 Dec  8 19:42 unbound_control.pem
-rw-r-----   1 unbound  unbound       2459 Dec  8 19:42 unbound_server.key
-rw-r-----   1 unbound  unbound       1318 Dec  8 19:42 unbound_server.pem

maybe it time to delete them, restart unbound or reboot pfsense.

themadsalvi · Mar 26, 2019, 7:03 PM

@RonpfS

what is the syntax for deleting the files in the shell?
rm -f /var/unbound/unbound_server.key?

is that the correct syntax?

Edit:
It looks like it was able to recreate the files
login-to-view

RonpfS · Mar 26, 2019, 8:06 PM

@themadsalvi

Rename them in case :

mv  /var/unbound/unbound_control.key /var/unbound/backup_unbound_control.key
mv  /var/unbound/unbound_control.pem /var/unbound/backup_unbound_control.pem
mv  /var/unbound/unbound_server.key /var/unbound/backup_unbound_server.key
mv  /var/unbound/unbound_server.pem /var/unbound/backup_unbound_server.pem

restart unbound, it should start, if not ... then move them back.
to remove them it's :

rm /var/unbound/unbound_server.pem

Also it's better to access the webgui with the pfsense IP address instead of using it's domain name when stopping and restarting DNS resolver.

themadsalvi · Mar 26, 2019, 7:08 PM

@RonpfS
unbound restarted ok, without any errors, but the DNSBL was still unable to reload without the error.
pfblockerng4.txt

I use the IP of Pfsense whenever I log into the web GUI, not sure why it uses the domain name when logging into shell

Grimson · Mar 26, 2019, 7:10 PM

What other packages are you using? Bind will conflict with unbound and if you use Service Watchdog make sure it does not monitor unbound.