DNS Resolver issues

smsigroupit

Good day!

We are experiencing frequent issues with our Netgate 8200 running:

pfSense Plus 26.03.1-RELEASE (amd64)

When the issue occurs, the pfSense web GUI becomes inaccessible, and all users lose internet access. We need to restart the pfSense appliance to restore both the web GUI and the internet connection.

We also found the following error in the system logs:
php-fpm 622 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1785105494] unbound[34372:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1785105494] unbound[34372:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1785105494] unbound[34372:0] error: bind: address already in use [1785105494] unbound[34372:0] fatal error: could not open ports'

Thank you.

Gertjan

The issue is :

@smsigroupit said in DNS Resolver issues:

error: bind: address already in use ..... fatal error: could not open ports

when a process, here unbound, attaches itself to ports (53 TCP and UDP) thse need to be 'free' or avaible. Only on process can attach to a port in "listen" mode.
Clear enough, another process is already listening. Yhis is most probably another instance of unbound, not yet terminated, or the process is in a zombie state.

Tell us more about your DNS setup :
You use the default resolving mode, or are you forwarding ?
Do you use pfBlocker ? (and if so, it better be 'python mode') And if so, how many DNSBL feeds ?
Do you use the system killer killer ? (Service_Watchdog).

edit : to get a stable system back fast :
Command line - option 8 (shell) and :

[26.03.1-RELEASE][root@pfSense.bhf.tld]/root: ps aux | grep 'unbound'
unbound 78638   0.0  2.6 131724 105792  -  Ss   08:07       0:06.32 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root    55152   0.0  0.3  23572  10984  -  S    16Jul26     0:23.48 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
root     4687   0.0  0.1  14196   2764  0  S+   08:18       0:00.00 grep unbound

Kill every process "/usr/local/sbin/unbound -c /var/unbound/unbound.conf".

For example,my case

kill 78638

and then , in the GUI, start it again.

Run

grep "start" /var/log/resolver.log

Does you 'unbound', the resolver, starts often ?
Ones or so per day is normal.
xx per minutes == very bad.

smsigroupit

@Gertjan

You use the default resolving mode, or are you forwarding ?
DNS Resolver

Do you use pfBlocker ? (and if so, it better be 'python mode') And if so, how many DNSBL feeds ?
pfBlocker was disabled.

Do you use the system killer killer ? (Service_Watchdog).
Im not using Service_Watchdog

Gertjan

@smsigroupit

You are using kea ? Or ISC ?

smsigroupit

Hi @Gertjan

Im using Kea

Gertjan

@smsigroupit

What does the 'system.log' show ?
More precise : what is triggering the resolver restart ? For example, if an interface (nay interface) goes down (and up again), many processes, and also unbound, restart.

smsigroupit

Hi @Gertjan

I try temporarily disable opt3 for monitoring.

Reply from Netgate:

Logs indicate that igc3 (OPT3) is flapping:

igc3: link state changed to UP
igc3: link state changed to DOWN

And , The logs also show repeated hotplug events for OPT3:

Jul 27 07:08:18 shoemakersfw php-fpm[98988]: NOTICE Hotplug event detected for IT (opt3) static IP ...............
Jul 27 07:08:18 shoemakersfw php-fpm[98988]: NOTICE DEVD Ethernet attached event for opt3
Jul 27 07:08:18 shoemakersfw php-fpm[98988]: NOTICE HOTPLUG: Triggering address refresh on opt3 (igc3)
Jul 27 07:08:18 shoemakersfw check_reload_status[686]: rc.newwanip starting igc3
Jul 27 07:08:18 shoemakersfw check_reload_status[686]: Reloading filter

Thank you for your help.

Gertjan

@smsigroupit said in DNS Resolver issues:

Reply from Netgate:

Logs indicate that igc3 (OPT3) is flapping:

igc3: link state changed to UP
igc3: link state changed to DOWN

From Netgate ?

An interface that goes down, also de-activated all processes that are listening (on a port) on that interface, they are all restarted, as you can see in the system log. Not only the resolver but also the web server (nginx) and others, like PHP, DHCP etc.
Flapping interfaces are fine for switches, they have nothing better to do, but I advise you strongly to adhere to this rule : never ever have interfaces going down (and up again) on routers (firewalls).
Your pfSense will thank you for this ^^

If the flapping goes fast, the the stopping and restarting goes faster as the process can shut down. Then you see the error you've shown at the start.

smsigroupit

@Gertjan

Thank you!