Unbound resolver error: Can't assign requested address for 127.0.0.1



  • Hello everyone,
    It appears as though an unbound resolver issue has hit my pfsense. I have tried pretty much everything that is simple. I even went as far as doing a factory reset, and setting everything up from a working configuration from last year. So far nothing has fixed the issue of this error popping up when I force reload DNSBL from pfblocker. The reason for the force reload is that the DNSBL is always out of sync, and sites will not load/resolve correctly. I have read another thread about this, and the user found that unbound_control.key and unbound_server.pem had been corrupted from a power outage. He was able to get them recreated, but did not post how. We had a power outage recently in my neighborhood right before this problem occurred.

    I wanted to know if there was a way to get pfsense to recreate these two files/keys in the GUI(getting to the console from there, etc)? I believe that @BBcan177 helped in this situation. Thank you for your time



  • Hi,

    Please confirm that you are using 2.4.4-p2.

    I advise you to isolate the problem.
    First : de activate packages like pfblocker.
    In the pfSEnse GUI : goto Status => Services and stop the Resolver.
    On another browser windows, visit Status > System > Logs > System > DNS Resolver
    In the first browser window, start the Resolver.
    In the second windows, hit Refesh (probably F5).

    Show us the log (reversed order) :

    Mar 26 15:46:59 	unbound 	12158:0 	info: start of service (unbound 1.8.1).
    ....
    Mar 26 15:46:30 	unbound 	13394:0 	info: service stopped (unbound 1.8.1).
    

    About the

    The manual states :

    # create certificate keys; do not recreate if they already exist.
    

    So, delete these two files (2 x .key file / 2 x .pem file) and restart unbound.

    Btw : this looks like a managed file system error. Certs are there, but can't be recreated - and contain wrong info.
    It's probably time to use the console access **, and execute a couple of times fsck.

    Power outage and the resulting hard shut down of pfSense ? Never saw that before.
    pfSense behaves like a Windows PC these days : pull out the plug and you have a good chance it won't wake up again.
    That's why UPS exist ... use them to remove a lot of possible problems.

    ** correction : if you do not use an UPS you should be an console access expert. That's one more reason why people prefer using an UPS ^^



  • @Gertjan
    these are the logs that I am able to get from the DNS resolver!resolver log 1.PNG dns resolver 2.PNG

    I will get a better UPS, as I have a 600VA one at the moment. It was not enough for this situation

    Edit: Placed the DNSBL reload error below for good measure

    DNSBL error.PNG

    I am in the shell, but unsure of commands to use to delete the files along with recreating them.



  • unbound listens to port 953 :

    [2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: sockstat -4l | grep 'unbound'
    unbound  unbound    28162 6  udp4   127.0.0.1:53          *:*
    unbound  unbound    28162 7  tcp4   127.0.0.1:53          *:*
    unbound  unbound    56848 6  udp4   *:53                  *:*
    unbound  unbound    56848 7  tcp4   *:53                  *:*
    unbound  unbound    56848 8  tcp4   127.0.0.1:953         *:*
    

    This :
    b578e5e5-1b45-4bfb-a6d9-c9db562a4443-image.png

    means probably that DNSBL tries to restart unbound, but it (re) started it to fast - unbound wasn't stopped (etc) and thus port '953' remains 'occupied'.
    The new unbound instance can't grab it - and complains about it after stopping.

    Run

    [2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ps ax | grep 'unbound'
    11558  -  Ss       0:00.22 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
    48881  1  S+       0:00.00 grep unbound
    

    to see what happens on your pfSEnse.
    If needed, stop unbound using the GUI, and if any zombies left, kill them.
    Using the kill command and the process number.

    edit : By any chance : your are not trying to overload unbound == very long startup time (by importing a huge number of DNSBL). On very small systems big lists can make unbound very slow to start, stop and just operate.



  • @Gertjan said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    unbound listens to port 953 :

    [2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: sockstat -4l | grep 'unbound'
    unbound  unbound    28162 6  udp4   127.0.0.1:53          *:*
    unbound  unbound    28162 7  tcp4   127.0.0.1:53          *:*
    unbound  unbound    56848 6  udp4   *:53                  *:*
    unbound  unbound    56848 7  tcp4   *:53                  *:*
    unbound  unbound    56848 8  tcp4   127.0.0.1:953         *:*
    

    This :
    b578e5e5-1b45-4bfb-a6d9-c9db562a4443-image.png

    means probably that DNSBL tries to restart unbound, but it (re) started it to fast - unbound wasn't stopped (etc) and thus port '953' remains 'occupied'.
    The new unbound instance can't grab it - and complains about it after stopping.

    Run

    [2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: ps ax | grep 'unbound'
    11558  -  Ss       0:00.22 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
    48881  1  S+       0:00.00 grep unbound
    

    to see what happens on your pfSEnse.
    If needed, stop unbound using the GUI, and if any zombies left, kill them.
    Using the kill command and the process number.

    edit : By any chance : your are not trying to overload unbound == very long startup time (by importing a huge number of DNSBL). On very small systems big lists can make unbound very slow to start, stop and just operate.

    This is before I disabled the resolver
    shell1.PNG

    After I disabled it, the grep command came up with this
    shell 2.PNG

    Then I killed the remaining 79387 process. The other process came up with a "no such process". Did I do this right? This comes up when doing the grep command after restarting the unbound resolver

    shell 3 after re-enable.PNG



  • @themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    DNSBL is always out of sync,

    Can you post the pfblockerng log during a Force Reload DNSBL so we can see why you get the Out of Sync errors ?

    You can get those errors when you have duplicate Headers / Label in DNSBL.

    How much memory do you have on that system ? 8GB can support around 1000000 DNSBL entries.



  • @Gertjan @RonpfS said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    @themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    DNSBL is always out of sync,

    Can you post the pfblockerng log during a Force Reload DNSBL so we can see why you get the Out of Sync errors ?

    You can get those errors when you have duplicate Headers / Label in DNSBL.

    How much memory do you have on that system ? 8GB can support around 1000000 DNSBL entries.

    I currently have 4GB, which is 45% percent used according to pfsense. I can add more, but I have been running it since last year with no issues.

    Below is a raw dump of the pfblockerng log in a text file(too many characters to do a full dump:

    pfblockerng.txt



  • Those tables : pfB_PRI1_v4, pfB_PRI4_v4, pfB_PRI2_v4, DNSBL_pfB_PRI2_v4 - pfB_PRI2_v4, DNSBL_Abuse - pfB_Abuse_PS_v4 shouldn't be in DNSBL, they are IPv4 tables, remove them.

    Disable BBC_DGA it's probably too big for your memory. And try another Force Reload DNSBL.

    You have to monitor memory usage with Status Monitoring. The Dashboard only display "current" memory usage, the Monitoring will give you memory usage over time.



  • @RonpfS

    Removed those, and forced a reload, which still had the unbound resolver error.

    This is the result in the status monitoring during and after reload
    a348ed5f-d979-4a1d-8682-09de6ce8d317-image.png

    This is the force reload log
    pfblockerng2.txt



  • You still have pfB_Abuse_PS_v4 to remove
    Try again with BBC_DGA feed disabled.
    If it still fails, then post your DNS Resolver config.



  • @RonpfS @Gertjan
    Here is the latest file for the reload, with all of the lists gone that you told me to delete. Same error pops up:
    pfblockerng3.txt

    cf0bfbbe-1751-4061-ad22-a07e5446cad1-image.png

    Rsolver settings.

    7a119d6b-0902-4162-b897-22902e3ce6d5-image.png

    211b1b7c-c022-4d22-874c-7cd89b024aa8-image.png

    b1b1b841-63b2-420c-ac63-134932daf8e6-image.png

    2a21e6fb-65bd-4ff2-a3cc-08c1069244f7-image.png

    eadfb9b5-9ce4-4979-a97c-683a4da03377-image.png



  • Did you try to remove the private-domain: line ?
    On my box I have Prefetch Support and Prefetch DNS Key Support ticked.



  • @RonpfS @Gertjan
    I ended up taking the private domain line out(save and apply), then checking the prefetch support and Prefetch DNS Key Support boxes(save and apply changes). Tried the forced reload, with those changes, and the error persists.



  • In a shell or Diagnostics Command prompt, do a

    ls -al /var/unbound /var/db/pfblockerng
    


  • @RonpfS @Gertjan
    I have placed the output below

    34befb63-1f5e-4954-9ce1-484201b12029-image.png
    afa1750a-be0a-4525-83fb-10a5ea7153e1-image.png

    Why are the last 4 so old?



  • @themadsalvi The 2012 timestamp looks suspicious compared to mine :

    -rw-r-----   1 unbound  unbound       2459 Dec  8 19:42 unbound_control.key
    -rw-r-----   1 unbound  unbound       1330 Dec  8 19:42 unbound_control.pem
    -rw-r-----   1 unbound  unbound       2459 Dec  8 19:42 unbound_server.key
    -rw-r-----   1 unbound  unbound       1318 Dec  8 19:42 unbound_server.pem
    
    

    maybe it time to delete them, restart unbound or reboot pfsense.



  • @RonpfS

    what is the syntax for deleting the files in the shell?
    rm -f /var/unbound/unbound_server.key?

    is that the correct syntax?

    Edit:
    It looks like it was able to recreate the files
    da3920eb-4780-4450-ab68-f3024e7e5c1d-image.png



  • @themadsalvi

    Rename them in case :

    mv  /var/unbound/unbound_control.key /var/unbound/backup_unbound_control.key
    mv  /var/unbound/unbound_control.pem /var/unbound/backup_unbound_control.pem
    mv  /var/unbound/unbound_server.key /var/unbound/backup_unbound_server.key
    mv  /var/unbound/unbound_server.pem /var/unbound/backup_unbound_server.pem
    

    restart unbound, it should start, if not ... then move them back.
    to remove them it's :

    rm /var/unbound/unbound_server.pem
    

    Also it's better to access the webgui with the pfsense IP address instead of using it's domain name when stopping and restarting DNS resolver.



  • @RonpfS
    unbound restarted ok, without any errors, but the DNSBL was still unable to reload without the error.
    pfblockerng4.txt

    I use the IP of Pfsense whenever I log into the web GUI, not sure why it uses the domain name when logging into shell



  • What other packages are you using? Bind will conflict with unbound and if you use Service Watchdog make sure it does not monitor unbound.



  • This post is deleted!


  • Well ... I have no more clue why it doesn't reload unbound.
    Maybe disable all feeds excepts Ads ?

    What does ls -al /var/unbound look like now ?



  • @RonpfS I placed the result of the rebuilt key and pem files, as well as how /var/unbound looks in my last post(out on lunch and on mobile, sorry)

    @Grimson the one thing I find odd is it just started this over the weekend, after a power outage. It has been fine for the last 6 months, without any issue. I do not have bind,and have made sure that unbound is not being monitored by service watchdog. I have the regularly installed packages like pfblockerng-devel, snort, etc.

    Screenshot_20190326-123300__01.jpg
    Screenshot_20190326-123308__01.jpg



  • @themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    @Grimson the one thing I find odd is it just started this over the weekend, after a power outage.

    So did you run fsck on the filesystem? https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-disk-check-errors-fsck.html#manually-run-fsck

    I have the regularly installed packages like pfblockerng-devel, snort, etc.

    There are no regularly installed packages, a regular install comes without additional packages. So always mention the packages you are using when asking for help.

    If following the above to check the filesystem doesn't work grab a config backup and do a fresh install to make sure the installation is in a good state.



  • @Grimson iplaced some screenshots from my mobile in my previouspost. Fsck says that /dev/zroot/ROOT cannot be opened since there is no file or directory present



  • @themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    @Grimson iplaced some screenshots from my mobile in my previouspost. Fsck says that /dev/zroot/ROOT cannot be opened since there is no file or directory present

    https://www.freebsd.org/doc/handbook/zfs-zpool.html#zfs-zpool-status
    https://www.freebsd.org/doc/handbook/zfs-zpool.html#zfs-zpool-scrub



  • @Grimson
    0dc2224a-ee6b-455c-abea-fef25d114303-image.png

    It looks like it found no errors in the pool. I even ran the scrub with no errors found.



  • To be honest, from what I can see your installation is a mess. For example you have both snort and suricata installed. Best suggestion is for you to document what you are currently using (and what not). Then start with a fresh clean install, don't restore the config do the setup bit by bit yourself and watch where it breaks.



  • @Grimson I will have to do that when I get home. I guess I have been lucky in that it has worked flawlessly for the last 6 months(SInce I installed it). It was bound to break eventually. Strangely, I can still surf the internet fine(I am writing from the network that has Pfsense firewall), and use streaming services like netflix.



  • You can always try to remove pfblocker lists by unticking pfBlockerNG & Keep Settings :

    Note: To clear all downloaded lists, uncheck these two checkboxes and 'Save'. Re-check both boxes and run a 'Force Update|Reload'



  • @RonpfS Uninstall, and then installed. This happened

    640c1eeb-d451-4313-9a1f-21bb52598302-image.png



  • Saw this :

    [ Windows_hosts_block ]		 Downloading update [ 03/26/19 12:04:08 ] .. 404 Not Found
    

    Possible to remove that feed ?
    (or am I to late already ?)



  • @themadsalvi So that is the output of a Force Update after Disabling pfblockerng and Save settings,
    Now when you tick both setttings, do a Force Update, then a Force Reload all.



  • @Gertjan said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    Possible to remove that feed ?

    Well the feed would be empty and would not prevent rebuilding the pfb_dnsbl.conf



  • @Gertjan I have started from zero on the pfblocker to try and rule that out.

    @RonpfS That is the output of fully removing and re-installing pfblocker, with the keep settings unticked.



  • @themadsalvi said in Unbound resolver error: Can't assign requested address for 127.0.0.1:

    @RonpfS That is the output of fully removing and re-installing pfblocker, with the keep settings unticked.

    Well let us know what happens when you add a DNSBL table.



  • @RonpfS If I do a force update and force reload with a list or two it does the following:

    UPDATE PROCESS START [ 03/26/19 14:48:53 ]

    ===[ DNSBL Process ]================================================

    [ 1 ] Downloading update .. 200 OK
    No Domains Found

    [ shalla ] Downloading update [ 03/26/19 14:49:03 ] .. 200 OK.

    Orig. Unique # Dups # White # Alexa Final

    19567 19567 0 0 0 19567

    DNSBL: Flush DNSBL_IP

    Assembling database... completed
    Validating database... Skipped
    Reloading Unbound... Failed to Reload... Restoring previous database.... Not completed.

    *** DNSBL update [ 0 ] [ 19567 ] ... OUT OF SYNC ! ***

    ===[ Continent Process ]============================================

    ===[ Aliastables / Rules ]==========================================

    No changes to Firewall rules, skipping Filter Reload
    No Changes to Aliases, Skipping pfctl Update

    ===[ FINAL Processing ]=====================================

    [ Original IP count ] [ 0 ]

    ===[ DNSBL Domain/IP Counts ] ===================================

    19567 /var/db/pfblockerng/dnsbl/shalla.txt

    IPv4 alias tables IP count

    1

    IPv6 alias tables IP count

    0

    Alias table IP Counts

       1 /var/db/aliastables/pfB_DNSBLIP.txt
    

    pfSense Table Stats

    table-entries hard limit 400000
    Table Usage Count 109283

    UPDATE PROCESS ENDED [ 03/26/19 14:49:07 ]



  • @RonpfS @Gertjan @Grimson thank you, and everyone for helping out. I ended up having to re-install pfsense again. I was able to find a config backup that worked, and allowed for unbound resolver to reload correctly. Unbound and DNS resolver is working correctly now. Thank you all so much!