Unbound crashes periodically with signal 11

jimp · Feb 19, 2021, 5:05 PM

@salander27-0 said in pfSense 2.50 snapshots have been dying for the past couple of days:

If there was a way for me to get a testing version of pfSense with Unbound 1.13.1 I would be more than happy to install that promptly and give feedback as to whether or not it is helpful at dealing with the issue.

We brought it in for snapshots (2.6.0 in the branch choices) but a new one hasn't built yet which includes it. In theory the branches are close enough at the moment you may be able to manually install the pkg archive file from the snapshot repo without much harm.

jimp · Feb 19, 2021, 5:34 PM

I edited the title of the thread to more accurately describe the issue.

It would also be helpful to know the hardware in the cases where this is happening (e.g. SG-3100, SG-5100, whitebox/custom hardware running CE, etc)

Salander27 0 · Feb 19, 2021, 6:51 PM

@jimp What is the repo URL for the snapshot repo that I can find that updated package in? I checked pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core and pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel and both still had unbound-1.13.0_2.

fivetoedslothbear · Feb 19, 2021, 7:34 PM

@jimp In my case, it's a custom Mini-ITX box I made with a Gigabyte B-150N motherboard (dual gigabit Intel NIC), and this is what the dashboard says about it:

CPU Type 	Intel(R) Celeron(R) CPU G3900 @ 2.80GHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: Yes (active)
Hardware crypto 	AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS
Kernel PTI 	Enabled

It's been running pfSense successfully for more years than I remember.

I'm getting occasional unbound crashes, and turned on the watchdog to restart the service when it dies.

ETA: I'm running the Community Edition.

jimp · Feb 19, 2021, 8:06 PM

@salander27-0 said in Unbound crashes periodically with signal 11:

@jimp What is the repo URL for the snapshot repo that I can find that updated package in? I checked pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core and pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel and both still had unbound-1.13.0_2.

In my previous reply I said "but a new one hasn't built yet which includes it." -- check later tonight/tomorrow AM.

e1219 · Feb 19, 2021, 8:11 PM

@jimp I am running pfSense CE inside a Proxmox (6.2-10) VM on a Qotom-Q555G6-S05 (i5 7200u).

login-to-view

I only installed the service watchdog package after this issue started occurring as suggested earlier on this thread. In the meantime, I have reverted to a backup of my VM pre-update running pfSense 2.4.5-1.

Salander27 0 · Feb 19, 2021, 9:02 PM

@jimp Sorry, I misunderstood what you saying. I'll check on a built package later.

Also, looks like people are posting on Reddit too.

hans109h · Feb 20, 2021, 1:07 AM

This was happening to me as well. I unchecked "DHCP registration" in the DNS Resolver config and for now it has eliminated the crash.

There was an issue before with this setting triggering an "unable to HUP" type error report, but I don't recall it causing a crash.

Fry-kun · Feb 20, 2021, 1:07 AM

I got tired of seeing delayed DNS queries (because watchdog doesn't restart the service immediately), so I'm currently running a bash loop:
while true; do /usr/local/sbin/unbound -vd -c /var/unbound/unbound.conf; done

-v makes Unbound print a message while starting, so I have a record of all restarts for the last 3 hours, here are the number of seconds between those (in case it helps):

249
213
1982
266
143
45
970
647
1312
4065
174
60

Doesn't seem to be consistent in my case -- but I also have an amount of devices on my network, maybe more devices makes things more noisy..?

Fry-kun · Feb 20, 2021, 1:32 AM

@jimp Do you still want more configuration examples?

Here's mine:
Packages:
Acme, Avahi, RRD Summary, Service_Watchdog, Shellcmd, System_Patches

server:
chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-glue: yes
do-ip4: yes
do-ip6: no
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 0
num-queries-per-thread: 512
jostle-timeout: 200
infra-host-ttl: 900
infra-cache-numhosts: 10000
outgoing-num-tcp: 10
incoming-num-tcp: 10
edns-buffer-size: 4096
cache-max-ttl: 86400
cache-min-ttl: 0
harden-dnssec-stripped: yes
msg-cache-size: 4m
rrset-cache-size: 8m
num-threads: 4
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
outgoing-range: 4096
auto-trust-anchor-file: /var/unbound/root.key
prefetch: no
prefetch-key: no
use-caps-for-id: no
serve-expired: no
aggressive-nsec: no
statistics-interval: 0
extended-statistics: yes
statistics-cumulative: yes
tls-cert-bundle: "/etc/ssl/cert.pem"
tls-port: 853
tls-service-pem: "/var/unbound/sslcert.crt"
tls-service-key: "/var/unbound/sslcert.key"
interface: 192.168.2.1
interface: 192.168.2.1@853
interface: 192.168.6.1
interface: 192.168.6.1@853
interface: 192.168.4.1
interface: 192.168.4.1@853
interface: 192.168.8.1
interface: 192.168.8.1@853
interface: fe80::201:2eff:fe78:9c5f%re1
interface: fe80::201:2eff:fe78:9c5f%re1@853
interface: fe80::201:2eff:fe78:9c5f%re1.6
interface: fe80::201:2eff:fe78:9c5f%re1.6@853
interface: fe80::201:2eff:fe78:9c5f%re1.4
interface: fe80::201:2eff:fe78:9c5f%re1.4@853
interface: fe80::201:2eff:fe78:9c5f%re1.8
interface: fe80::201:2eff:fe78:9c5f%re1.8@853
interface: 127.0.0.1
interface: 127.0.0.1@853
interface: ::1
interface: ::1@853
outgoing-interface: <*** REDACTED ***>
outgoing-interface: <*** REDACTED ***>
private-address: 127.0.0.0/8
private-address: 10.0.0.0/8
private-address: ::ffff:a00:0/104
private-address: 172.16.0.0/12
private-address: ::ffff:ac10:0/108
private-address: 169.254.0.0/16
private-address: ::ffff:a9fe:0/112
private-address: 192.168.0.0/16
private-address: ::ffff:c0a8:0/112
private-address: fd00::/8
private-address: fe80::/10
include: /var/unbound/access_lists.conf
include: /var/unbound/host_entries.conf
include: /var/unbound/dhcpleases_entries.conf
include: /var/unbound/domainoverrides.conf
forward-zone:
	name: "."
	forward-addr: 8.8.8.8
	forward-addr: 8.8.4.4
server:
 log-servfail: yes
 private-domain: "pfsense.mydomain.com"
include: /var/unbound/remotecontrol.conf

both "Register DHCP leases in the DNS Resolver" and "Register DHCP static mappings in the DNS Resolver" are enabled

Lease time is default (24hrs, I think?)

jkv · Feb 20, 2021, 6:00 AM

@hans109h

I also had the “ Register DHCP leases in the DNS Resolver” option enabled when seeing this issue and disabling this option appears to have stopped the crash.

maverick_slo · Feb 20, 2021, 6:02 AM

@jkv +1
No packages at all...

Salander27 0 · Feb 20, 2021, 7:20 AM

@fry-kun I suspect you're seeing crashes approximately whenever certain devices renew their DHCP lease. Since the timing of this has more to do with when that device was turned on you would expect to see somewhat random delays between such renewals.

Question for those experiencing these crashes, do you have both "Register DHCP static mappings in the DNS Resolver" enabled AND DHCP Static Mappings where at least one mapping has something filled in in the hostname field?

Fry-kun · Feb 20, 2021, 7:27 AM

@salander27-0 said in Unbound crashes periodically with signal 11:

Question for those experiencing these crashes, do you have both "Register DHCP static mappings in the DNS Resolver" enabled AND DHCP Static Mappings where at least one mapping has something filled in in the hostname field?

Affirmative

tuxbox78 · Feb 20, 2021, 12:59 PM

@salander27-0 Yes, same here on my system..both were enabled and have quite some manual mappings in my system. After upgrade to 2.5 it seems unbound crashed about every ten minutes, after disabling register DHCP mapping it stopped crashing.

My System is based on a standalone AMD GX-412TC SOC having run pfSense stable for years now.

e1219 · Feb 20, 2021, 2:44 PM

@salander27-0 I can confirm this as well, I have this option set with a handful of static mappings with hostname specified.

jkv · Feb 20, 2021, 3:11 PM

@salander27-0

confirmed

Salander27 0 · Feb 22, 2021, 8:50 PM

@jimp I have successfully downloaded and installed unbound-1.13.1 from the devel repo now that the package has been built. No crashes yet, but none of the DHCP leases of my static leases (with hostname) have expired yet so I wouldn't expect crashes yet. I will update in a few hours once those leases start expiring.

OBSOLETE PLEASE SEE https://forum.netgate.com/post/966915

For anyone else who wishes to try unbound 1.13.1 (normal caveats about this being unsupported and at your own risk) I simplified the install command to the following:

pkg add -f https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/All/unbound-1.13.1.txz

You can roll back with:

pkg install -f unbound

After either of these commands you will need to restart the unbound service to pick up on the new binary.

OBSOLETE PLEASE SEE https://forum.netgate.com/post/966915

Fry-kun · Feb 20, 2021, 9:42 PM

@salander27-0 been running 1.13.1 for ~2hrs, no crashes yet!
1.13.0 crashed way more often, as evidenced in my earlier message

Salander27 0 · Feb 20, 2021, 9:43 PM

Alright, it's been about 2 and a half hours since I installed unbound-1.13.1 on my system. I have stress-tested it by reducing the DHCP lease time to 120 seconds and have since seen hundreds of DHCP renewals (and subsequent unbound HUPs). Were this unbound 1.13.0 I would have likely seen dozens of crashes, however unbound 1.13.1 has been completely stable in that time.

At this point I believe the best course of option is for additional people to test to see if their issue is resolved with the updated version. If you are willing to take system backups and accept the (IMO very low) risk that you may need to reinstall pfSense completely if something goes wrong (so don't do this on your prod systems) then please follow the instructions in my previous comment to install unbound-1.13.1.