DNS randomly stops working

[NickH 0]

Hi All

Basic setup - clients have their DNS set to the netgate router LAN / Gateway.

Clients will randomly stop working and cant access anything on the internet. setting a client to 1.1.1.1 DNS manually and voila it works. a reboot of the netgate works and it will work for a while and then out of no where clients using the netgate as DNS will stop working again. Its getting annoying now and wondering if anyone has experienced this?

[plfinch]

I have been experiencing this as well every few days since upgrading an SG-5100 to 21.02 (and now p1). Unbound silently dies and stops logging, and pfSense shows unbound as not running. This never happened previously with 2.4.5p1 and prior. I just setup some probes and monitors yesterday to see if I can learn more at next occurrence.

[LuigiM81]

Same here with 2.5 CE

After restart unbound via Services everything works great again.

Mario

[bingo600]

@luigim81

I'm using the service watchdog package , to monitor & restart unbound.
If it crashes ...

/Bingo

[NickH 0]

@bingo600 is this in the package maanger ?

[bingo600]

@nickh-0

Yes

login-to-view

Note : I'm still on 2.4.5-p1 , but i'd be suprised if it isn't in 2.5 too.

/Bingo

[Gertjan]

Hummm.

Try using 2.5.0 - Unbound never failed (crashed) on me.
edit : neither the version before.

Btw : I'm using a 'vanilla' intel based box (Dell desktop PC stripped down).

login-to-view

Uncheck that one if you don't want unbound to get restart after each DHCP lease coming in .....

"pfBlockerNG 3" doesn't like that option neither :

login-to-view

[SashLi]

Same issue here but on 2.5.1. The DNS Server stopped working. The strange thing is that netstat is showing that port 23 is bounded and the gui showing that the service is running but there is no way to get a DNS resolve via the localhost server ( already checked bound interfaces ). Have a box with the same configuration on 2.5.0 and that is working correctly. It is also working if I'm using a external DNS server only with setting to ignore the local

[Gertjan]

@sashli said in DNS randomly stops working:

The strange thing is that netstat is showing that port 23 is bounded

That's a telnet server. Not DNS related.

@sashli said in DNS randomly stops working:

and the gui showing that the service is running

Do this extra test:

ps ax | grep `cat /var/run/unbound.pid`

If it shows :

xxxxxx-  Ss      5:23.38 /usr/local/sbin/unbound -c /var/unbound/unbound.conf

You will know for sure that unbound is there.

sockstat | grep 'unbound'

or

sockstat | grep '53'

shows you on which interfaces it lists.
"netstat" is more a linux command. It's "sockstat" for FreeBSD.

For me, it listens to all interfaces (no exceptions) :

unbound unbound 6660 3 udp6 *:53 :
unbound unbound 6660 4 tcp6 *:53 :
unbound unbound 6660 5 udp4 *:53 :
unbound unbound 6660 6 tcp4 *:53 :
unbound unbound 6660 7 tcp4 127.0.0.1:953 :

and this for TCP and UDP, IPv4 and IPv6.
This includes localhost == the old 127.0.0.1 or thew ::1.

The '953' is the special control access, which is only IPv4 - localhost.

[2.5.0-RELEASE][admin@pfsense.mylocal.net]/var/run: dig @::1 pfsense.mylocal.net. +short
192.168.1.1

This means that unbound is up, running, and answering.

Also : 2.5.1 has some unbound related updates/upgrades.
I installed and tested it @home since yesterday. It's as good as 2.5.0 (and in theory better ^^)

@sashli said in DNS randomly stops working:

Have a box with the same configuration on 2.5.0 and that is working correctly.

This means : compare the two (only the settings you changed from default) and you'll have your answer.

[vjizzle]

Unbound on 2.5 CE crashed regularly for me also. For the brief period that I was on 2.5.1 CE unbound did not crash. I did find that when I upgrade unbound pfSense 2.5 it is not crashing anymore. You can do this from the Command Prompt or shell access.

The command: pkg upgrade -fy unbound

Installed packages to be UPGRADED:
	unbound: 1.13.0_2 -> 1.13.1 [pfSense]

Make sure to have backups before you do this :). I did find that this makes unbound stable on pfSense 2.5 CE for me but YMMV.

[SashLi]

@gertjan same result. host is showing that unbound is up and working but not resolving.

ps ax | grep `cat /var/run/unbound.pid`


12809  -  Is      0:00.02 /usr/local/sbin/unbound -c /var/unbound/unbound.conf

sockstat report running service

unbound  unbound    36677 4  udp6   *:53                  *:*
unbound  unbound    36677 5  tcp6   *:53                  *:*
unbound  unbound    36677 6  udp4   *:53                  *:*
unbound  unbound    36677 7  tcp4   *:53                  *:*
unbound  unbound    36677 8  tcp4   127.0.0.1:953         *:*

Doing a dig on local now lead to

dig @::1 pfsense.org +short
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address
208.123.73.69

the stange thing is when trying to do this against the IPv4 localhost 127.0.0.1

dig @127.0.0.1 pfsense.org +short
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

; <<>> DiG 9.16.12 <<>> @127.0.0.1 pfsense.org +short
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Also trying the GUI DNS Test fail with no response. I added the ::1 to test if it is a problem resolving against 127.0.0.1 only

login-to-view

[juniper]

@sashli same problem here:

after upgrading pfsense from 2.5.0 to 2.5.1 unboud stop responding on 127.0.0.1:

[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: dig @127.0.0.1
net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

[2.5.1-RELEASE][admin@ids.*********.**]/etc/defaults: ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address
ping: sendto: Can't assign requested address

unbound service starts:

[2.5.1-RELEASE][admin@ids.********.**]/etc/defaults: netstat -ln
Active Internet connections
Proto Recv-Q Send-Q Local Address                                 Foreign Address                               (state)    
tcp4       0      0 10.10.200.2.443                               10.10.15.3.56532                              ESTABLISHED
tcp4       0     44 10.10.200.2.22                                10.10.15.3.37490                              ESTABLISHED
tcp4       0      0 10.10.200.2.22                                10.10.15.3.37488                              ESTABLISHED
udp6       0      0 ::1.53                                        *.*                                           
udp4       0      0 127.0.0.1.53                                  *.*                                           

On pfsense firewall we have snort and pfblockerNG, nothing is changed from 2.5.0, we have only upgraded pfsense.

Some suggestion?

[Gertjan]

@sashli said in DNS randomly stops working:

dig @127.0.0.1 pfsense.org +short

Let's add some details.
Lot's of details.

dig @127.0.0.1 pfsense.org +trace

But the real issue is here :

net.c:536: probing sendmsg() with IP_TOS=b8 failed: Can't assign requested address

'dig' is not family of, or related to 'unbound'.
dig comes from the bind FreeBSD package 'bind-tools'.

"net.c" is one of the main source code files, and it can not "use the IP you gave" ... it can not use "127.0.0.1" so it can contact unbound over 127.0.0.1 port 53 UDP.

When I stop unbound - so no one is listing on 127.0.0.1 port 53, and execute a
dig @127.0.0.1 pfsense.org +trace
it comes back after several seconds with a logical "connection timed out; no servers could be reached"

You have interface problems ..... and not 'unbound' problems. unbound is yet another victim of the real issue.

[juniper]

@gertjan yes problem is related to interfaces...

if i try to ping 127.0.0.1 i've the same error...

Someone opened a bug for that?

[SashLi]

@gertjan thanks for your feedback but it looks really like a problem with the localhost 127.0.0.1 interface. I just reinstalled the ntopng and this has also problems starting and running services. Issue leading in the same direction here "Could not connect to Redis at 127.0.0.1:6379: Can't assign requested address'".

The question is: why is the interface on the localhost not reachable by the system itself ? Any idea ?

[juniper]

@sashli try to ping other interfaces from pfsense itself...

i think is jail related or something similar...

[SashLi]

@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem

IPv4 link-local (169.254.x.x) gateway does not function

[juniper]

@sashli i don't know...

I can add an information, my pfsense installation is on proxmox ve virtual machine.

[Gertjan]

@sashli said in DNS randomly stops working:

@juniper just found this ticket in redmine, not sure if this can be an issue created by this gatewax problem

IPv4 link-local (169.254.x.x) gateway does not function

That's a special case, using IP's like 169.254.0.0/16.
There is a small patch for this - jimp posted one yesterday.

I've updated my pfSense at home (Hyper-V = VM) based : it's just perfect.
I've just updated my work pfSense, they update went just fine. Again perfect.

127.0.0.1 isn't even an external driver related, as it is part of the build in 'kernel' IP stack facilities.

Use another VM host - if you have a Windows 10 Pro somewhere, you could make one right away, or install pfSense bare bone and you see that their is - can not be - localhost issues, as it will break everything.

Just to be sure : your issue exists after you reset to default - and you only changed the password (!! did NOT import your settings !!) ?

[juniper]

@sashli

Problem appear if i insert 127.0.0.1 on GENERAL SETUP---DNS Server Settings

I use to have there 127.0.0.1 if pfsense is a dns resolver.

If i config with 127.0.0.1 GENERAL SETUP---DNS Server Settings as a default pfsense dns server there is the problem with interface lo0.