Unbound exited signal 11

popcorn9499

Recently had to setup local dns and therefore started to use pfsense's dns server to manage that. Anyways since then I've been getting a random crash every day or two where unbound fails causing me not to be able to resolve domain names temporarily.

Sep 4 06:49:01	kernel		cannot forward src fe80:1::62e9:ab02:3fc1:72c1, dst 2605:b100:280:a::26, nxt 58, rcvif igb0, outif igb1
Sep 4 01:37:21	kernel		pid 12320 (unbound), jid 0, uid 59: exited on signal 11
Sep 4 01:01:07	php	2002	rc.dyndns.update: phpDynDNS (home): (Unknown Response)

and then today I had the issue again

Sep 6 16:50:41 	php-fpm 	56950 	/index.php: Successful login for user 'admin' from: 192.168.1.195 (Local Database)
Sep 6 16:09:09 	kernel 		pid 7413 (unbound), jid 0, uid 59: exited on signal 11
Sep 6 15:10:36 	kernel 		cannot forward src fe80:1::45f6:b4d9:b8de:2437, dst 2605:b100:280:a::6, nxt 58, rcvif igb0, outif igb1

I looked in other logs and saw nothing however next time it occurs I will definitely add more information to this post.

I am running pfsense 2.6.0-RELEASE.
running 4gb of ram with a intel core 2 6420 cpu. (I know its old but I had it kicking around and it seems to work and my memory usage never seems to go above 30%

packages installed:
Bandwidthd 0.7.4_5
Service_Watchdog 1.8.7_1 (Installed to work around the unbound issue)
Status Traffic Totals 2.3.2_2

I have nothing fancy enabled in my dns settings. I have dnssec enabled but besides that everything else is disabled. it isn't even set to do dhcp registration.

Unsure what the best course of action is.
Thank you for your time.

Gertjan

@popcorn9499 said in Unbound exited signal 11:

exited on signal 11

Is a segmentation fault. That's not known at all for a 2.6.0.

Can you post logs lines above the "exited on signal 11" ?

You've read : Unbound crashes periodically with signal 11
That forum thread talks about 2.5.0 unbound version 1.12.x ?, 1.13.x ?

There is an ongoing issue with 22.05 for some people as it is not using the same version for unbound as 2.6.0 - pfSense running on native, bare bone 'old' Intel CPU : I never had any issues. neither using Hyper-V @home.

Be careful with Service_Watchdog.
It's very possible that unbound is restarted for some interface event like a changing WAN IP.
The unbound restart takes 'some time', like a couple of seconds or so, could be a bit more.
But Service_Watchdog isn't patient. It checks the PID, doesn't see the process (it was just restarting) so it fires the resurrection bullet. Unbound getting restarted during restart == no good.
The unbound version used by 2.6.0 is pretty stable, but again, that my IMHO.

Can you post your /var/unbound/unbound.conf file ?
Do you use 'out of the box' settings ? Or are you forwarding ?

What happens when you remove these very resources intensive packages : Status Traffic Totals and Bandwidthd ?

@popcorn9499 said in Unbound exited signal 11:

it isn't even set to do dhcp registration

You've been reading the forum.

keyser

@popcorn9499 Agree with @Gertjan
I had severe issues with Unbound becoming unstable and buggy because of Watchdog attempting to restart it in situations where it would/needed to reload itself for various reasons (DHCP client registration, pFBlockerNG updates before python mode).

popcorn9499

@keyser noted I ended up removing watchdog as I wanna debug this and find the proper solution.

popcorn9499

@gertjan said in Unbound exited signal 11:

@popcorn9499 said in Unbound exited signal 11:

exited on signal 11

Is a segmentation fault. That's not known at all for a 2.6.0.

Can you post logs lines above the "exited on signal 11"

system.log
if you want any other logs let me know

You've read : Unbound crashes periodically with signal 11
That forum thread talks about 2.5.0 unbound version 1.12.x ?, 1.13.x ?

Ya I was reading those but figured it was worth mentioning regardless. I wasn't seeing on 2.6.0 just 2.5

There is an ongoing issue with 22.05 for some people as it is not using the same version for unbound as 2.6.0 - pfSense running on native, bare bone 'old' Intel CPU : I never had any issues. neither using Hyper-V @home.

I don't even believe this has any hypervisor extensions enabled however its definitely running bare metal with a intel i340-T2 nic specifically a ibm 49Y232 according to the ebay listing.

Be careful with Service_Watchdog.
It's very possible that unbound is restarted for some interface event like a changing WAN IP.
The unbound restart takes 'some time', like a couple of seconds or so, could be a bit more.
But Service_Watchdog isn't patient. It checks the PID, doesn't see the process (it was just restarting) so it fires the resurrection bullet. Unbound getting restarted during restart == no good.

I have now uninstalled Service watchdog. If I don't need it its one less thing to go wrong

The unbound version used by 2.6.0 is pretty stable, but again, that my IMHO.

Can you post your /var/unbound/unbound.conf file ?
it seemed longer than what should be put in a forum
Do you use 'out of the box' settings ? Or are you forwarding ?

my pfsense installation is mostly out of the box. I have some vlans setup and some host overrides in my dns settings as well. I do have some upnp configuration since this is setup for a home network and being able to have upnp work for games was somewhat important.
the only other thing I have touched was lowing the number of hosts to cache to see if maybe it was some sort of issue where it was running lower on memory. However that hasn't fixed the issue and it was happening prior as well. I also have dynamic dns configured however it broke and I just hadn't gotten around to fixing it.

Not sure what forwarding is. if you mean port forwarding then yes I have a fair number of ports forwarded.

What happens when you remove these very resources intensive packages : Status Traffic Totals and Bandwidthd ?
I will update you guys in a day or two. Removed those packages just a few moments or so and it has been happening every few days so that may fix it.

Thank you so much for spending the time to read this I honestly wasn't too sure where to start or what information to provide

Gertjan

@popcorn9499 said in Unbound exited signal 11:

I also have dynamic dns configured however it broke and I just hadn't gotten around to fixing i

That issue is written in the logs "Passwords do not match".

cannot forward src fe80:1::45f6:b4d9:b8de:2437, dst 2605:b100:280

Are you using IPv6 ?
Who is fe80:1::45f6:b4d9:b8de:2437 ?

If your not using ipv6, add this to the unbound custom config ()

server :
       do-ip6: no

as it won't hurt.
Strange is, I've been using 2.6.0 for a year or more, and use IPv4 and IPv6.
Never saw unbound failing.

Sep 6 16:09:09 MegaLaptop kernel: pid 7413 (unbound), jid 0, uid 59: exited on signal 11

The log line before was on Sep 6 15:10:36, that's one hour before the unbound crash.
So nothing really related in the system log.

Sep 6 16:58:49 MegaLaptop php-fpm[56261]: /rc.start_packages: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found'
Sep 6 16:58:49 MegaLaptop vnstatd[6435]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.

These two lines are really strange.
You've said you have bandwidthd and Traffic Totals installed ( is "vnstat" related to "Traffic Totals", I'm not sure, I'm not using both packages).
They already died ? If there is a pid file, and no related process bandwidthd running, that's a bad sign.

Installed packages are restarted, as pfSense 'though' they are running.but both this wasn't the case.

@popcorn9499 said in Unbound exited signal 11:

with a intel i340-T2 nic

.... and a realtek NIC. Thats always a red flag. Maybe not this time.
Disabling a realtek is always a good thing to do.

Can you count the number of unbound start per day , week ?
Use :

grep 'start' /var/log/resolver.log

Just to check if it isn't restart xxx times a day.