DNS randomly stops working
-
Unbound on 2.5 CE crashed regularly for me also. For the brief period that I was on 2.5.1 CE unbound did not crash. I did find that when I upgrade unbound pfSense 2.5 it is not crashing anymore. You can do this from the Command Prompt or shell access.
The command: pkg upgrade -fy unbound
Installed packages to be UPGRADED: unbound: 1.13.0_2 -> 1.13.1 [pfSense]Make sure to have backups before you do this :). I did find that this makes unbound stable on pfSense 2.5 CE for me but YMMV.
-
@gertjan same result. host is showing that unbound is up and working but not resolving.
ps ax | grep `cat /var/run/unbound.pid` 12809 - Is 0:00.02 /usr/local/sbin/unbound -c /var/unbound/unbound.confsockstat report running service
unbound unbound 36677 4 udp6 *:53 *:* unbound unbound 36677 5 tcp6 *:53 *:* unbound unbound 36677 6 udp4 *:53 *:* unbound unbound 36677 7 tcp4 *:53 *:* unbound unbound 36677 8 tcp4 127.0.0.1:953 *:*Doing a dig on local now lead to
dig @::1 pfsense.org +short net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address 208.123.73.69the stange thing is when trying to do this against the IPv4 localhost 127.0.0.1
dig @127.0.0.1 pfsense.org +short net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address ; <<>> DiG 9.16.12 <<>> @127.0.0.1 pfsense.org +short ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reachedAlso trying the GUI DNS Test fail with no response. I added the ::1 to test if it is a problem resolving against 127.0.0.1 only
-
@sashli same problem here:
after upgrading pfsense from 2.5.0 to 2.5.1 unboud stop responding on 127.0.0.1:
[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: dig @127.0.0.1 net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address[2.5.1-RELEASE][admin@ids.*********.**]/etc/defaults: ping 127.0.0.1 PING 127.0.0.1 (127.0.0.1): 56 data bytes ping: sendto: Can't assign requested address ping: sendto: Can't assign requested address ping: sendto: Can't assign requested address ping: sendto: Can't assign requested addressunbound service starts:
[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: netstat -ln Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 10.10.200.2.443 10.10.15.3.56532 ESTABLISHED tcp4 0 44 10.10.200.2.22 10.10.15.3.37490 ESTABLISHED tcp4 0 0 10.10.200.2.22 10.10.15.3.37488 ESTABLISHED udp6 0 0 ::1.53 *.* udp4 0 0 127.0.0.1.53 *.*On pfsense firewall we have snort and pfblockerNG, nothing is changed from 2.5.0, we have only upgraded pfsense.
Some suggestion?
-
@sashli said in DNS randomly stops working:
dig @127.0.0.1 pfsense.org +short
Let's add some details.
Lot's of details.dig @127.0.0.1 pfsense.org +traceBut the real issue is here :
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address'dig' is not family of, or related to 'unbound'.
dig comes from the bind FreeBSD package 'bind-tools'."net.c" is one of the main source code files, and it can not "use the IP you gave" ... it can not use "127.0.0.1" so it can contact unbound over 127.0.0.1 port 53 UDP.
When I stop unbound - so no one is listing on 127.0.0.1 port 53, and execute a
dig @127.0.0.1 pfsense.org +trace
it comes back after several seconds with a logical "connection timed out; no servers could be reached"You have interface problems ..... and not 'unbound' problems. unbound is yet another victim of the real issue.
-
@gertjan yes problem is related to interfaces...
if i try to ping 127.0.0.1 i've the same error...
Someone opened a bug for that?
-
@gertjan thanks for your feedback but it looks really like a problem with the localhost 127.0.0.1 interface. I just reinstalled the ntopng and this has also problems starting and running services. Issue leading in the same direction here "Could not connect to Redis at 127.0.0.1:6379: Can't assign requested address'".
The question is: why is the interface on the localhost not reachable by the system itself ? Any idea ?
-
@sashli try to ping other interfaces from pfsense itself...
i think is jail related or something similar...
-
@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem
-
@sashli i don't know...
I can add an information, my pfsense installation is on proxmox ve virtual machine.
-
@sashli said in DNS randomly stops working:
@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem
That's a special case, using IP's like 169.254.0.0/16.
There is a small patch for this - jimp posted one yesterday.I've updated my pfSense at home (Hyper-V = VM) based : it's just perfect.
I've just updated my work pfSense, they update went just fine. Again perfect.127.0.0.1 isn't even an external driver related, as it is part of the build in 'kernel' IP stack facilities.
Use another VM host - if you have a Windows 10 Pro somewhere, you could make one right away, or install pfSense bare bone and you see that their is - can not be - localhost issues, as it will break everything.
Just to be sure : your issue exists after you reset to default - and you only changed the password (!! did NOT import your settings !!) ?
-
Problem appear if i insert 127.0.0.1 on GENERAL SETUP---DNS Server Settings
I use to have there 127.0.0.1 if pfsense is a dns resolver.
If i config with 127.0.0.1 GENERAL SETUP---DNS Server Settings as a default pfsense dns server there is the problem with interface lo0.
-
@juniper Then you create a loopback??
If nothing is in General setup then it uses localhost.
-
@cool_corona no i use 127.0.0.1 as dns resolver for pfsense.
-
-
@cool_corona yes it works!
Problem appear if i use 127.0.0.1 in general setup.
If i set an external dns and set dns resolution behavior as you suggest all working fine.
Thank you.
-
@juniper exactly its a problem when 127.0.0.1 is listed in the general setup of the DNS server list.
-
Why do you want to add 127.0.0.1 here :
?
As you can see, I have nothing.
Because that's default : nothing.Still, the magic is happening :
I hope (didn't test) that pfSense is intelligent enough that, when 127.0.0.1 is added here :
It will ignore that 'request' as 127.0.0.1 is already there.
Here it is :
/etc/resolv.conf .......nameserver 127.0.0.1 search your-domain.tldOk, I broke my won rules
and added some DNS settings myself.
but /etc/resolv.conf didn't change.
I'm missing something ....But pfSense (unbound) works.edit : this is a no go :
[2.5.1-RELEASE][admin@pfsense.my-domain.tld]/root: dig @127.0.0.1 google.com
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address
net.c:536: probing sendmsg() with IPV6_TCLASS=b8 failed: No route to hostOk. Great.
Who calls Houston ?Solution : remove ::1 and 127.0.0.1 from the General settings as it was
Useless (before)
Break things (today).So, please : don't do that ;)
edit : even why I removed ::1 and 127.0.0.1 I had could not use "127.0.0.1" any more.
It was :
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested addressall the time now.
Si, guys,I don't quiet understand what I'm seeing, but I see what you see.
Unbound wasn't listening to 127.0.0.1 any more - I restarted unbound : didn't help. I had to restart "127.0.0.1" - if possible.
I had to reboot pfSense - as this is a way to 'restart' the kernel. -
@gertjan Personally had 127.0.0.1 from before that "DNS Resolution Behavior" section existed, a couple of years at least.
If memory serves right
Guess that's because we wanted to ensure the system used its own resolver, and that only.Was stunned when all DNS resolution stopped after upgrading to 2.5.1. Not even pkg worked :)
But some dig(ging) lead to the entry removal, then restoration of DNS service.No bug listed as of now that I can find, but @jimp could we have this case covered in the pfsense-upgrade script that gets updated when pressing "13" in the CLI? It would eliminate the 127.0.0.1 entry, as workaround for now in place of debugging the lovely
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested addressbork message.
-
I am also having this problem after upgrading directory from 2.4.5 CE to 2.5.1 on a Sophos SG-210.
In my case, enabling unbound on the Service Watchdog list restarts the service, but then the CPU is pegged at 100% and resolution still doesn't happen. Restarting the firewall works. I have not yet checked the PID or socket status during an outage, but I suspect unbound crashes, thinks its still running but can't clean itself up.One thing I noticed on my system is pkg info unbound says Python is enabled, even though it is disabled in the configuration. I manually restarted after toggling Python on and off. Is this even relevant?
[2.5.1-RELEASE][admin@myfirewallnotyours]/root: pkg info unbound unbound-1.13.1 Name : unbound Version : 1.13.1 Installed on : Thu Apr 15 03:10:26 2021 CDT Origin : dns/unbound Architecture : FreeBSD:12:amd64 Prefix : /usr/local Categories : dns Licenses : BSD3CLAUSE Maintainer : jaap@NLnetLabs.nl WWW : https://www.nlnetlabs.nl/projects/unbound Comment : Validating, recursive, and caching DNS resolver Options : DNSCRYPT : off DNSTAP : off DOCS : off DOH : on ECDSA : on EVAPI : off FILTER_AAAA : off GOST : on HIREDIS : off LIBEVENT : on MUNIN_PLUGIN : off PYTHON : on SUBNET : off TFOCL : off TFOSE : off THREADS : on Shared Libs required: libexpat.so.1 libnghttp2.so.14 libpython3.7m.so.1.0 libevent-2.1.so.7 Shared Libs provided: libunbound.so.8 Annotations : FreeBSD_version: 1202504 cpe : cpe:2.3:a:nlnetlabs:unbound:1.13.1:::::freebsd12:x64 repo_type : binary repository : pfSense Flat size : 7.79MiBTried all recommendations on this post but nothing is working so far.
-
Eagerly following any threads about DNS, My watch dog is restarting Unbound all the time.
2.5.1-RELEASE (amd64)
pfBlockerNG-devel: 3.0.0_16
snort: 4.1.3_5
Telegraf: 0.9_5Just wanted to share what see just on the off chance it helps the group, I did notice the already in use error in my system logs, when the watch dog is trying to start it back up
Apr 23 10:15:04 pfsense php[92161]: servicewatchdog_cron.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1619136904] unbound[68018:0] debug: creating udp4 socket 192.168.1.1 53 [1619136904] unbound[68018:0] error: bind: address already in use [1619136904] unbound[68018:0] fatal error: could not open ports' Apr 23 10:15:04 pfsense php[73303]: notify_monitor.php: Message sent to XXXXX@hotmail.com OK Apr 23 10:15:01 pfsense php[92161]: servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)Brisbane Queensland Australia
