2.5.0 DNS Service stopping randomly

noviceiii

Dear all

I have the same issue after updating to 2.5.0.
The unbound service stopps randomly after several times per hour.

I use "DNS forwarder" module mostly on default settings. I have increased DNS chache to 10 MB and set particular ingoing and outgoing network interfaces. Beside that no changes to the previous pfsense version.

There isn't much in the logs... All what I can see from syslog every time it stopps is below output. The only thing I can directly relate to is the kernel message at 15:53:37. However, I see the same messages at every restart.

Feb 19 15:53:06	arpwatch	90833	bogon 0.0.0.0 70:xx:xx:xx:xx:xx
Feb 19 15:53:37	kernel		pid 4137 (unbound), jid 0, uid 59: exited on signal 11
Feb 19 15:54:00	php	45674	servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)

As a temporary workaorund I have installed watchdog and added unbound to be restarted automatically. Further, I'll put logging to Level 5 and would report back here if I can find more information.

Any hint to resolve this issue would be welcome.

Kind regards,
Oliver

noviceiii

@yis said in 2.5.0 DNS Service stopping randomly:

Hi there,
..

How can I activate the debug log for unbound - was not able to find this on the first view..

you can change the log level in Services -> DNS Resolver -> Advanced Settings -> "Log Level", 5th last from the bottom.

swampkracker

I am having the same problem. For now, I am using DNS forwarder until the issue with Unbound is fixed.

noviceiii

Dear all

Since I have Loglevel 5 enabled, I now get below log entries on unbound service terminations:

Feb 19 16:36:24	php-fpm	384	/services_unbound_advanced.php: The command '/usr/sbin/arp -s '192.168.my.pfsense-server' '00:0c:xx:xx:xx:xx'' returned exit code '1', the output was 'arp: writing to routing socket: Operation not permitted'
Feb 19 16:42:55	kernel		pid 15151 (unbound), jid 0, uid 59: exited on signal 11
Feb 19 16:43:00	php	23895	servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)

...hmm

Salander27 0

FYI this is a dupe of https://forum.netgate.com/topic/160005/unbound-crashes-periodically-with-signal-11/31

CCNewb

@yis known issue 100s are experiencing. Right now only good solution is to untick "Register DHCP leases in the DNS Resolver" and unbound will stop crashing.

CCNewb

@salander27-0 said in 2.5.0 DNS Service stopping randomly:

FYI this is a dupe of https://forum.netgate.com/topic/160005/unbound-crashes-periodically-with-signal-11/31

what a worthless comment. "Here go read 50 replies and waste 20 minutes until you find the solution"...how about just give the solution and not point to some other massively long thread.

Salander27 0

@ccnewb At this point there is no "solution". There are several workarounds (disabling "Register DHCP leases in the DNS Resolver" counts as a workaround, not a solution) but the exact cause is still unknown. There has been more troubleshooting and analysis in the other thread and a Netgate dev there is actively soliciting more information so the cause can be identified. There is little point in having multiple threads going about the same issue and duplicating effort to identify the cause.

Also, you should strive to be a bit more friendly when responding in this forum. pfSense-CE is community-supported software after all.

Kartoff

I had same issue when I did and update to 2.5.0... At first I didn't even noticed there is a problem... Soon after couple of my friends call me to tell they have problem with internet... Finally I saw unbound was stopped somehow and couple hours after started it sops again... Finally I put it in service watchdog and there are no problem anymore since then...

noviceiii

Thank you all for the responses. I did investigate a bit further on the ARP messages that caught my attention shortly before the unbound services crashes. To my surprise I couldn't find a device with the particular hardware address among my devices.

So I flushed ARP cache and DHCP cache. Now I have working non-crashing unbound service.

SOLUTION:

FLUSH DHCP LEASES

Allow me to add my 10 cents about the watchdog workaround I have used too. A device directly connected to the internet should never have crashing services that are automatically restarted. It could be a huge security hole. Further, on any system, to fix a permanently restarting service by automatically starting it is just pure wrong,

I like to mention, that the DNS resolver with pfsense 2.5.0 feels much faster than it was with the previous version. Thank you, good peolple from Netgate!

yis

Wow - I am off for two days and this topic was growing great!
Thank you all for your ideas and supportings.
I will check the point with DHCP FLUSH this evening and monitor this the next days

yis

ok - I flushed DHCP leases last night - this morning the unbound service is dead again.
Going to start debug logs now, will report back...

EDIT: this is the error I found in logs..

debug:    ip4 10.1.1.1 port 53 (len 16)
debug: attempt to get extra 3 targets
debug: servselect ip4 10.1.1.1 port 53 (len 16)
debug:    rtt=120000
debug: No more query targets, attempting last resort
debug: configured stub or forward servers failed -- returning SERVFAIL
debug: store error response in message cache
debug: return error response SERVFAIL
debug: mesh_run: iterator module exit state is module_finished

will try to figure out, what this means

Traveller

@yis Take a look at this suggestion.

This new version of unbound is reportedly more stable based on the initial testing.

noviceiii

after flushing the DHCP leases I had ubound service running for about 3 days until it started to crash again. So unfortunately flushing the DHCP leases isn't - depending on your systems load - a long lasting solution (although I like the word 'flush').

I did follow the instructions given by the great and honorable member @jimp (see Posting) to update the ubound service. I'll keep monitoring but will report back only, if I face further crashes.

jimps proposed solution was:

Go to Diagnostics > Command Prompt

pkg upgrade -fy unbound; pfSsh.php playback svc restart unbound

yis

@noviceiii said in 2.5.0 DNS Service stopping randomly:

after flushing the DHCP leases I had ubound service running for about 3 days until it started to crash again. So unfortunately flushing the DHCP leases isn't - depending on your systems load - a long lasting solution (although I like the word 'flush').

I did follow the instructions given by the great and honorable member @jimp (see Posting) to update the ubound service. I'll keep monitoring but will report back only, if I face further crashes.

jimps proposed solution was:

Go to Diagnostics > Command Prompt
pkg upgrade -fy unbound; pfSsh.php playback svc restart unbound

flushing dhcp is not the solution.
and I don't want to write down every single dns from dhcp-lease - so i will upgrade unbound now and will have a look on this.

yis

I can confirm problem as resolved after upgrading unbound.
working stable again - thank you for all you support!

JasonAU

@yis said in 2.5.0 DNS Service stopping randomly:

I can confirm problem as resolved after upgrading unbound.
working stable again - thank you for all you support!

Thanks for this, I am getting 100's of e-mail from Service Watchdog

Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)

Browsing is often hit & miss Chrome will was unable to resolve and then once the service starts again load the page, I'll backup and see how a unbound upgrade goes for reference and anyone else trying this mine did

unbound: 1.13.0_2 -> 1.13.1 [pfSense]

CCNewb

the new unbound is not resolving the issue :/

Mar 14 13:23:40 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).
Mar 14 13:23:46 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).
Mar 14 13:29:28 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).
Mar 14 13:34:54 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).
Mar 14 13:36:11 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).
Mar 14 13:36:22 unbound 33517 [33517:0] info: service stopped (unbound 1.13.1).

JasonAU

@ccnewb Yep I'm also in the same boat
I"ve uninstalled PFblocker and made sure register in DNS when DHCP lease is not checked same deal