DNS randomly stops working

vjizzle

Unbound on 2.5 CE crashed regularly for me also. For the brief period that I was on 2.5.1 CE unbound did not crash. I did find that when I upgrade unbound pfSense 2.5 it is not crashing anymore. You can do this from the Command Prompt or shell access.

The command: pkg upgrade -fy unbound

Installed packages to be UPGRADED:
	unbound: 1.13.0_2 -> 1.13.1 [pfSense]

Make sure to have backups before you do this :). I did find that this makes unbound stable on pfSense 2.5 CE for me but YMMV.

SashLi

@gertjan same result. host is showing that unbound is up and working but not resolving.

ps ax | grep `cat /var/run/unbound.pid`


12809  -  Is      0:00.02 /usr/local/sbin/unbound -c /var/unbound/unbound.conf

sockstat report running service

unbound  unbound    36677 4  udp6   *:53                  *:*
unbound  unbound    36677 5  tcp6   *:53                  *:*
unbound  unbound    36677 6  udp4   *:53                  *:*
unbound  unbound    36677 7  tcp4   *:53                  *:*
unbound  unbound    36677 8  tcp4   127.0.0.1:953         *:*

Doing a dig on local now lead to

dig @::1 pfsense.org +short
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address
208.123.73.69

the stange thing is when trying to do this against the IPv4 localhost 127.0.0.1

dig @127.0.0.1 pfsense.org +short
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

; <<>> DiG 9.16.12 <<>> @127.0.0.1 pfsense.org +short
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Also trying the GUI DNS Test fail with no response. I added the ::1 to test if it is a problem resolving against 127.0.0.1 only

DNS issue.png

juniper

@sashli same problem here:

after upgrading pfsense from 2.5.0 to 2.5.1 unboud stop responding on 127.0.0.1:

[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: dig @127.0.0.1
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

[2.5.1-RELEASE][admin@ids.*********.**]/etc/defaults: ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address

unbound service starts:

[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: netstat -ln
Active Internet connections
Proto Recv-Q Send-Q Local Address                                 Foreign Address                               (state)    
tcp4       0      0 10.10.200.2.443                               10.10.15.3.56532                              ESTABLISHED
tcp4       0     44 10.10.200.2.22                                10.10.15.3.37490                              ESTABLISHED
tcp4       0      0 10.10.200.2.22                                10.10.15.3.37488                              ESTABLISHED
udp6       0      0 ::1.53                                        *.*                                           
udp4       0      0 127.0.0.1.53                                  *.*

On pfsense firewall we have snort and pfblockerNG, nothing is changed from 2.5.0, we have only upgraded pfsense.

Some suggestion?

Gertjan

@sashli said in DNS randomly stops working:

dig @127.0.0.1 pfsense.org +short

Let's add some details.
Lot's of details.

dig @127.0.0.1 pfsense.org +trace

But the real issue is here :

net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

'dig' is not family of, or related to 'unbound'.
dig comes from the bind FreeBSD package 'bind-tools'.

"net.c" is one of the main source code files, and it can not "use the IP you gave" ... it can not use "127.0.0.1" so it can contact unbound over 127.0.0.1 port 53 UDP.

When I stop unbound - so no one is listing on 127.0.0.1 port 53, and execute a
dig @127.0.0.1 pfsense.org +trace
it comes back after several seconds with a logical "connection timed out; no servers could be reached"

You have interface problems ..... and not 'unbound' problems. unbound is yet another victim of the real issue.

juniper

@gertjan yes problem is related to interfaces...

if i try to ping 127.0.0.1 i've the same error...

Someone opened a bug for that?

SashLi

@gertjan thanks for your feedback but it looks really like a problem with the localhost 127.0.0.1 interface. I just reinstalled the ntopng and this has also problems starting and running services. Issue leading in the same direction here "Could not connect to Redis at 127.0.0.1:6379: Can't assign requested address'".

The question is: why is the interface on the localhost not reachable by the system itself ? Any idea ?

juniper

@sashli try to ping other interfaces from pfsense itself...

i think is jail related or something similar...

SashLi

@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem

IPv4 link-local (169.254.x.x) gateway does not function

juniper

@sashli i don't know...

I can add an information, my pfsense installation is on proxmox ve virtual machine.

Gertjan

@sashli said in DNS randomly stops working:

@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem

IPv4 link-local (169.254.x.x) gateway does not function

That's a special case, using IP's like 169.254.0.0/16.
There is a small patch for this - jimp posted one yesterday.

I've updated my pfSense at home (Hyper-V = VM) based : it's just perfect.
I've just updated my work pfSense, they update went just fine. Again perfect.

127.0.0.1 isn't even an external driver related, as it is part of the build in 'kernel' IP stack facilities.

Use another VM host - if you have a Windows 10 Pro somewhere, you could make one right away, or install pfSense bare bone and you see that their is - can not be - localhost issues, as it will break everything.

Just to be sure : your issue exists after you reset to default - and you only changed the password (!! did NOT import your settings !!) ?

juniper

@sashli

Problem appear if i insert 127.0.0.1 on GENERAL SETUP---DNS Server Settings

I use to have there 127.0.0.1 if pfsense is a dns resolver.

If i config with 127.0.0.1 GENERAL SETUP---DNS Server Settings as a default pfsense dns server there is the problem with interface lo0.

Cool_Corona

@juniper Then you create a loopback??

If nothing is in General setup then it uses localhost.

juniper

@cool_corona no i use 127.0.0.1 as dns resolver for pfsense.

Cool_Corona

@juniper Leave it blank and

And reboot

juniper

@cool_corona yes it works!

Problem appear if i use 127.0.0.1 in general setup.

If i set an external dns and set dns resolution behavior as you suggest all working fine.

Thank you.

SashLi

@juniper exactly its a problem when 127.0.0.1 is listed in the general setup of the DNS server list.

Gertjan

@juniper

Why do you want to add 127.0.0.1 here :

?

As you can see, I have nothing.
Because that's default : nothing.

Still, the magic is happening :

I hope (didn't test) that pfSense is intelligent enough that, when 127.0.0.1 is added here :

It will ignore that 'request' as 127.0.0.1 is already there.
Here it is :
/etc/resolv.conf .......

nameserver 127.0.0.1
search your-domain.tld

Ok, I broke my won rules and added some DNS settings myself.

but /etc/resolv.conf didn't change.
I'm missing something ....

~~But pfSense (unbound) works.~~

edit : this is a no go :

[2.5.1-RELEASE][admin@pfsense.my-domain.tld]/root: dig @127.0.0.1 google.com
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address
net.c:536: probing sendmsg() with IPV6_TCLASS=b8 failed: No route to host

Ok. Great.
Who calls Houston ?

Solution : remove ::1 and 127.0.0.1 from the General settings as it was
Useless (before)
Break things (today).

So, please : don't do that ;)

edit : even why I removed ::1 and 127.0.0.1 I had could not use "127.0.0.1" any more.

It was :

net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

all the time now.

Si, guys,I don't quiet understand what I'm seeing, but I see what you see.

Unbound wasn't listening to 127.0.0.1 any more - I restarted unbound : didn't help. I had to restart "127.0.0.1" - if possible.
I had to reboot pfSense - as this is a way to 'restart' the kernel.

e-1-1

@gertjan Personally had 127.0.0.1 from before that "DNS Resolution Behavior" section existed, a couple of years at least.
If memory serves right
Guess that's because we wanted to ensure the system used its own resolver, and that only.

Was stunned when all DNS resolution stopped after upgrading to 2.5.1. Not even pkg worked :)
But some dig(ging) lead to the entry removal, then restoration of DNS service.

No bug listed as of now that I can find, but @jimp could we have this case covered in the pfsense-upgrade script that gets updated when pressing "13" in the CLI? It would eliminate the 127.0.0.1 entry, as workaround for now in place of debugging the lovely

net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

bork message.

collinatselect

I am also having this problem after upgrading directory from 2.4.5 CE to 2.5.1 on a Sophos SG-210.
In my case, enabling unbound on the Service Watchdog list restarts the service, but then the CPU is pegged at 100% and resolution still doesn't happen. Restarting the firewall works. I have not yet checked the PID or socket status during an outage, but I suspect unbound crashes, thinks its still running but can't clean itself up.

One thing I noticed on my system is pkg info unbound says Python is enabled, even though it is disabled in the configuration. I manually restarted after toggling Python on and off. Is this even relevant?

[2.5.1-RELEASE][admin@myfirewallnotyours]/root: pkg info unbound
unbound-1.13.1
Name           : unbound
Version        : 1.13.1
Installed on   : Thu Apr 15 03:10:26 2021 CDT
Origin         : dns/unbound
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : dns
Licenses       : BSD3CLAUSE
Maintainer     : jaap@NLnetLabs.nl
WWW            : https://www.nlnetlabs.nl/projects/unbound
Comment        : Validating, recursive, and caching DNS resolver
Options        :
        DNSCRYPT       : off
        DNSTAP         : off
        DOCS           : off
        DOH            : on
        ECDSA          : on
        EVAPI          : off
        FILTER_AAAA    : off
        GOST           : on
        HIREDIS        : off
        LIBEVENT       : on
        MUNIN_PLUGIN   : off
        PYTHON         : on
        SUBNET         : off
        TFOCL          : off
        TFOSE          : off
        THREADS        : on
Shared Libs required:
        libexpat.so.1
        libnghttp2.so.14
        libpython3.7m.so.1.0
        libevent-2.1.so.7
Shared Libs provided:
        libunbound.so.8
Annotations    :
        FreeBSD_version: 1202504
        cpe            : cpe:2.3:a:nlnetlabs:unbound:1.13.1:::::freebsd12:x64
        repo_type      : binary
        repository     : pfSense
Flat size      : 7.79MiB

Tried all recommendations on this post but nothing is working so far.

JasonAU

Eagerly following any threads about DNS, My watch dog is restarting Unbound all the time.

2.5.1-RELEASE (amd64)
pfBlockerNG-devel: 3.0.0_16
snort: 4.1.3_5
Telegraf: 0.9_5

Just wanted to share what see just on the off chance it helps the group, I did notice the already in use error in my system logs, when the watch dog is trying to start it back up

Apr 23 10:15:04 pfsense php[92161]: servicewatchdog_cron.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1619136904] unbound[68018:0] debug: creating udp4 socket 192.168.1.1 53 [1619136904] unbound[68018:0] error: bind: address already in use [1619136904] unbound[68018:0] fatal error: could not open ports'
Apr 23 10:15:04 pfsense php[73303]: notify_monitor.php: Message sent to XXXXX@hotmail.com OK
Apr 23 10:15:01 pfsense php[92161]: servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)