• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Unbound crashes periodically with signal 11

2.5 Development Snapshots (Retired)
36
105
30.5k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mkernalcon
    last edited by Feb 18, 2021, 9:25 PM

    After updating just a few hours ago to the 2.5.0 release on our main router, I can confirm that I am having the same issue. I've temporarily fixed it by disabling the "Register DHCP leases in the DNS Resolver" option. I can confirm, looking through the logs, that several HUPs get sent properly, all to the same PID, before finally starting to fail with "No such process". The last HUP that doesn't immediately fail in DHCP logs, has exactly the same timestamp as "pid 55598 (unbound), jid 0, uid 59: exited on signal 11" in the general log. No information in the DNS resolver logs.

    DHCP Leases are default (7200s) for all vlans except my "main" lan, which is 691200s. Looks like the HUPs that kill it come from the 7200s vlans, but this is probably just coincidence.

    Installed Packages:
    darkstat
    iperf
    nmap
    nut
    openvpn-client-export
    Status_Traffic_Totals

    unbound.conf (has not been edited manually):

    ##########################
    # Unbound Configuration
    ##########################
    
    ##
    # Server configuration
    ##
    server:
    
    chroot: /var/unbound
    username: "unbound"
    directory: "/var/unbound"
    pidfile: "/var/run/unbound.pid"
    use-syslog: yes
    port: 53
    verbosity: 2
    hide-identity: yes
    hide-version: yes
    harden-glue: yes
    do-ip4: yes
    do-ip6: no
    do-udp: yes
    do-tcp: yes
    do-daemonize: yes
    module-config: "validator iterator"
    unwanted-reply-threshold: 0
    num-queries-per-thread: 512
    jostle-timeout: 200
    infra-host-ttl: 900
    infra-cache-numhosts: 10000
    outgoing-num-tcp: 10
    incoming-num-tcp: 10
    edns-buffer-size: 4096
    cache-max-ttl: 86400
    cache-min-ttl: 0
    harden-dnssec-stripped: yes
    msg-cache-size: 4m
    rrset-cache-size: 8m
    
    num-threads: 12
    msg-cache-slabs: 8
    rrset-cache-slabs: 8
    infra-cache-slabs: 8
    key-cache-slabs: 8
    outgoing-range: 4096
    #so-rcvbuf: 4m
    auto-trust-anchor-file: /var/unbound/root.key
    prefetch: no
    prefetch-key: no
    use-caps-for-id: no
    serve-expired: no
    aggressive-nsec: no
    # Statistics
    # Unbound Statistics
    statistics-interval: 0
    extended-statistics: yes
    statistics-cumulative: yes
    
    # TLS Configuration
    tls-cert-bundle: "/etc/ssl/cert.pem"
    
    # Interface IP(s) to bind to
    interface: 192.168.2.1
    interface: 192.168.3.1
    interface: 192.168.4.1
    interface: 192.168.11.1
    interface: 192.168.99.1
    interface: 127.0.0.1
    interface: ::1
    
    # Outgoing interfaces to be used
    
    # DNS Rebinding
    # For DNS Rebinding prevention
    private-address: 127.0.0.0/8
    private-address: 10.0.0.0/8
    private-address: ::ffff:a00:0/104
    private-address: 172.16.0.0/12
    private-address: ::ffff:ac10:0/108
    private-address: 169.254.0.0/16
    private-address: ::ffff:a9fe:0/112
    private-address: 192.168.0.0/16
    private-address: ::ffff:c0a8:0/112
    private-address: fd00::/8
    private-address: fe80::/10
    # Set private domains in case authoritative name server returns a Private IP address
    
    
    
    # Access lists
    include: /var/unbound/access_lists.conf
    
    # Static host entries
    include: /var/unbound/host_entries.conf
    
    # dhcp lease entries
    include: /var/unbound/dhcpleases_entries.conf
    
    
    
    # Domain overrides
    include: /var/unbound/domainoverrides.conf
    
    
    
    
    ###
    # Remote Control Config
    ###
    include: /var/unbound/remotecontrol.conf
    
    1 Reply Last reply Reply Quote 0
    • J
      jimp Rebel Alliance Developer Netgate
      last edited by Feb 18, 2021, 9:54 PM

      OK so nothing jumps out in those configs. I still can't make it crash here even hammering on it.

      I see that Unbound 1.13.1 is out now, we might need to pull that in and test against it. I reopened https://redmine.pfsense.org/issues/11316 which was initially closed since we didn't have enough information.

      Keep the details coming here on this forum post, we may still be able to spot a pattern.

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      M 1 Reply Last reply Feb 19, 2021, 2:48 AM Reply Quote 0
      • M
        mxw39 @jimp
        last edited by mxw39 Feb 19, 2021, 2:53 AM Feb 19, 2021, 2:48 AM

        @jimp thanks for the support! Is there some debug command that helps collect logs? dumps syscalls before segfault? I hope to contribute my crashing unbound somehow.

        1 Reply Last reply Reply Quote 0
        • J
          jkv
          last edited by jkv Feb 19, 2021, 1:50 PM Feb 19, 2021, 1:49 PM

          After upgrading two systems (one to CE 2.5.0 and the other, running negate hardware (SG-5100) , to + 21.02) I have also started seeing this issue.

          Just in case it helps in spotting any common patterns - I note that:

          • On both systems (between every 5 to 10 minutes) I see unbound stopping and restarting in the DNS Resolver log.
          • Only on SG-5100 (pfsense + 21.02) I am also seeing (probably 8 times a day or so) in the General System log that unbound exited on signal 11 (for example "pid 73090 (unbound), jid 0, uid 59: exited on signal 11").
          • The only packages installed in common on both systems are Cron (0.3.7_5), openvpn-client-export (1.5_5) and Service_Watchdog (1.8.7_1).
          • The SG-5100 (pfsense + 21.02) also has installed arpwatch (0.2.0_4), freeradius3 (0.15.7_29), and pfBlockerNG-devel (3.0.0_10).
          • Both systems have WAN connections with dynamic IPs (so ddns is in use on the WAN side).
          • Both systems also have some Static DHCP entries set (with "Register DHCP static mappings in the DNS Resolver" enabled).
          G 1 Reply Last reply Feb 19, 2021, 2:17 PM Reply Quote 0
          • G
            Gertjan @jkv
            last edited by Feb 19, 2021, 2:17 PM

            @jkv said in pfSense 2.50 snapshots have been dying for the past couple of days:

            I see unbound stopping

            &

            @jkv said in pfSense 2.50 snapshots have been dying for the past couple of days:

            General System log that unbound exited on signal 11

            You see it dying.
            You use Service_Watchdog to restart it - right ?

            @jkv said in pfSense 2.50 snapshots have been dying for the past couple of days:

            pfBlockerNG-devel (3.0.0_10).

            How often is the pfBlockerNG-devel doing it's cron task ? This task is logged. Does it restart unbound ?
            What happens when you stop "Service_Watchdog ", so it doesn't restart unbound ?

            What I'm trying to find out : if Service_Watchdog detects that unbound stops, it launches another instance. But it was actually just stopping and restarting, ordered by pfBlockerNG-devel. So, two instances are started, one dies .....
            This is just a theory, as I'm not using Service_Watchdog myself

            Also, SG-5100 is an Intel based machine, so "You and I" are using the same executable / same binary. Only our "config" differs. I don't know nothing about ARM based binaries, but I tend to say the "Intel" ones are pretty solid.

            These :

            @jkv said in pfSense 2.50 snapshots have been dying for the past couple of days:

            have some Static DHCP entries

            do nothing to unbound. The "static DHCP settings" (host name IP relation) are copied in the /etc/hosts file during boot. this file is (also) read by unbound during it's initial start up. These 'static DHCP setting' rarely change, that is, only if you delete/modify/add one. Look at this file, you'll see what I mean.
            (In the past) the "DHCP Registration / Register DHCP leases in the DNS Resolver" could be problematic. The ""static DHCP settings"" were never a source of issue.

            No "help me" PM's please. Use the forum, the community will thank you.
            Edit : and where are the logs ??

            J 1 Reply Last reply Feb 19, 2021, 2:35 PM Reply Quote 0
            • J
              jkv @Gertjan
              last edited by Feb 19, 2021, 2:35 PM

              @gertjan

              the cron job for pfBlockerNG-devel is hourly and there does not appear to be any correlation between this cron job and unbound exiting. I will do some testing with Service_Watchdog disabled to see what happens to unbound.

              S 1 Reply Last reply Feb 27, 2021, 5:44 AM Reply Quote 1
              • S
                Salander27 0
                last edited by Feb 19, 2021, 4:59 PM

                I doubt that service watchdog is the cause of the issue. It wasn't even present on my installation until I installed it so I wouldn't have to manually restart unbound after the crashes.

                1 Reply Last reply Reply Quote 0
                • S
                  Salander27 0
                  last edited by Feb 19, 2021, 5:03 PM

                  If there was a way for me to get a testing version of pfSense with Unbound 1.13.1 I would be more than happy to install that promptly and give feedback as to whether or not it is helpful at dealing with the issue.

                  Also, can we get the title of this forum post updated to something like "DNS Resolver/Unbound crashing on pfSense 2.5" so that we can attract the attention of anyone else searching for this issue?

                  J 1 Reply Last reply Feb 19, 2021, 5:05 PM Reply Quote 0
                  • J
                    jimp Rebel Alliance Developer Netgate @Salander27 0
                    last edited by Feb 19, 2021, 5:05 PM

                    @salander27-0 said in pfSense 2.50 snapshots have been dying for the past couple of days:

                    If there was a way for me to get a testing version of pfSense with Unbound 1.13.1 I would be more than happy to install that promptly and give feedback as to whether or not it is helpful at dealing with the issue.

                    We brought it in for snapshots (2.6.0 in the branch choices) but a new one hasn't built yet which includes it. In theory the branches are close enough at the moment you may be able to manually install the pkg archive file from the snapshot repo without much harm.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • J
                      jimp Rebel Alliance Developer Netgate
                      last edited by Feb 19, 2021, 5:34 PM

                      I edited the title of the thread to more accurately describe the issue.

                      It would also be helpful to know the hardware in the cases where this is happening (e.g. SG-3100, SG-5100, whitebox/custom hardware running CE, etc)

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      F E 2 Replies Last reply Feb 19, 2021, 7:33 PM Reply Quote 0
                      • S
                        Salander27 0
                        last edited by Feb 19, 2021, 6:51 PM

                        @jimp What is the repo URL for the snapshot repo that I can find that updated package in? I checked pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core and pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel and both still had unbound-1.13.0_2.

                        J 1 Reply Last reply Feb 19, 2021, 8:06 PM Reply Quote 1
                        • F
                          fivetoedslothbear @jimp
                          last edited by fivetoedslothbear Feb 19, 2021, 7:34 PM Feb 19, 2021, 7:33 PM

                          @jimp In my case, it's a custom Mini-ITX box I made with a Gigabyte B-150N motherboard (dual gigabit Intel NIC), and this is what the dashboard says about it:

                          CPU Type 	Intel(R) Celeron(R) CPU G3900 @ 2.80GHz
                          2 CPUs: 1 package(s) x 2 core(s)
                          AES-NI CPU Crypto: Yes (active)
                          Hardware crypto 	AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS
                          Kernel PTI 	Enabled
                          

                          It's been running pfSense successfully for more years than I remember.

                          I'm getting occasional unbound crashes, and turned on the watchdog to restart the service when it dies.

                          ETA: I'm running the Community Edition.

                          1 Reply Last reply Reply Quote 0
                          • J
                            jimp Rebel Alliance Developer Netgate @Salander27 0
                            last edited by Feb 19, 2021, 8:06 PM

                            @salander27-0 said in Unbound crashes periodically with signal 11:

                            @jimp What is the repo URL for the snapshot repo that I can find that updated package in? I checked pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core and pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel and both still had unbound-1.13.0_2.

                            In my previous reply I said "but a new one hasn't built yet which includes it." -- check later tonight/tomorrow AM.

                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            S 1 Reply Last reply Feb 20, 2021, 7:17 PM Reply Quote 1
                            • E
                              e1219 @jimp
                              last edited by Feb 19, 2021, 8:11 PM

                              @jimp I am running pfSense CE inside a Proxmox (6.2-10) VM on a Qotom-Q555G6-S05 (i5 7200u).

                              login-to-view

                              I only installed the service watchdog package after this issue started occurring as suggested earlier on this thread. In the meantime, I have reverted to a backup of my VM pre-update running pfSense 2.4.5-1.

                              1 Reply Last reply Reply Quote 1
                              • S
                                Salander27 0
                                last edited by Feb 19, 2021, 9:02 PM

                                @jimp Sorry, I misunderstood what you saying. I'll check on a built package later.

                                Also, looks like people are posting on Reddit too.

                                1 Reply Last reply Reply Quote 0
                                • H
                                  hans109h
                                  last edited by Feb 19, 2021, 10:09 PM

                                  This was happening to me as well. I unchecked "DHCP registration" in the DNS Resolver config and for now it has eliminated the crash.

                                  There was an issue before with this setting triggering an "unable to HUP" type error report, but I don't recall it causing a crash.

                                  F J 2 Replies Last reply Feb 20, 2021, 1:07 AM Reply Quote 0
                                  • F
                                    Fry-kun @hans109h
                                    last edited by Feb 20, 2021, 1:07 AM

                                    I got tired of seeing delayed DNS queries (because watchdog doesn't restart the service immediately), so I'm currently running a bash loop:
                                    while true; do /usr/local/sbin/unbound -vd -c /var/unbound/unbound.conf; done

                                    -v makes Unbound print a message while starting, so I have a record of all restarts for the last 3 hours, here are the number of seconds between those (in case it helps):

                                    249
                                    213
                                    1982
                                    266
                                    143
                                    45
                                    970
                                    647
                                    1312
                                    4065
                                    174
                                    60

                                    Doesn't seem to be consistent in my case -- but I also have an amount of devices on my network, maybe more devices makes things more noisy..?

                                    S 1 Reply Last reply Feb 20, 2021, 7:20 AM Reply Quote 0
                                    • F
                                      Fry-kun @jimp
                                      last edited by Fry-kun Feb 20, 2021, 1:32 AM Feb 20, 2021, 1:30 AM

                                      @jimp Do you still want more configuration examples?

                                      Here's mine:
                                      Packages:
                                      Acme, Avahi, RRD Summary, Service_Watchdog, Shellcmd, System_Patches

                                      server:
                                      chroot: /var/unbound
                                      username: "unbound"
                                      directory: "/var/unbound"
                                      pidfile: "/var/run/unbound.pid"
                                      use-syslog: yes
                                      port: 53
                                      verbosity: 1
                                      hide-identity: yes
                                      hide-version: yes
                                      harden-glue: yes
                                      do-ip4: yes
                                      do-ip6: no
                                      do-udp: yes
                                      do-tcp: yes
                                      do-daemonize: yes
                                      module-config: "validator iterator"
                                      unwanted-reply-threshold: 0
                                      num-queries-per-thread: 512
                                      jostle-timeout: 200
                                      infra-host-ttl: 900
                                      infra-cache-numhosts: 10000
                                      outgoing-num-tcp: 10
                                      incoming-num-tcp: 10
                                      edns-buffer-size: 4096
                                      cache-max-ttl: 86400
                                      cache-min-ttl: 0
                                      harden-dnssec-stripped: yes
                                      msg-cache-size: 4m
                                      rrset-cache-size: 8m
                                      num-threads: 4
                                      msg-cache-slabs: 4
                                      rrset-cache-slabs: 4
                                      infra-cache-slabs: 4
                                      key-cache-slabs: 4
                                      outgoing-range: 4096
                                      auto-trust-anchor-file: /var/unbound/root.key
                                      prefetch: no
                                      prefetch-key: no
                                      use-caps-for-id: no
                                      serve-expired: no
                                      aggressive-nsec: no
                                      statistics-interval: 0
                                      extended-statistics: yes
                                      statistics-cumulative: yes
                                      tls-cert-bundle: "/etc/ssl/cert.pem"
                                      tls-port: 853
                                      tls-service-pem: "/var/unbound/sslcert.crt"
                                      tls-service-key: "/var/unbound/sslcert.key"
                                      interface: 192.168.2.1
                                      interface: 192.168.2.1@853
                                      interface: 192.168.6.1
                                      interface: 192.168.6.1@853
                                      interface: 192.168.4.1
                                      interface: 192.168.4.1@853
                                      interface: 192.168.8.1
                                      interface: 192.168.8.1@853
                                      interface: fe80::201:2eff:fe78:9c5f%re1
                                      interface: fe80::201:2eff:fe78:9c5f%re1@853
                                      interface: fe80::201:2eff:fe78:9c5f%re1.6
                                      interface: fe80::201:2eff:fe78:9c5f%re1.6@853
                                      interface: fe80::201:2eff:fe78:9c5f%re1.4
                                      interface: fe80::201:2eff:fe78:9c5f%re1.4@853
                                      interface: fe80::201:2eff:fe78:9c5f%re1.8
                                      interface: fe80::201:2eff:fe78:9c5f%re1.8@853
                                      interface: 127.0.0.1
                                      interface: 127.0.0.1@853
                                      interface: ::1
                                      interface: ::1@853
                                      outgoing-interface: <*** REDACTED ***>
                                      outgoing-interface: <*** REDACTED ***>
                                      private-address: 127.0.0.0/8
                                      private-address: 10.0.0.0/8
                                      private-address: ::ffff:a00:0/104
                                      private-address: 172.16.0.0/12
                                      private-address: ::ffff:ac10:0/108
                                      private-address: 169.254.0.0/16
                                      private-address: ::ffff:a9fe:0/112
                                      private-address: 192.168.0.0/16
                                      private-address: ::ffff:c0a8:0/112
                                      private-address: fd00::/8
                                      private-address: fe80::/10
                                      include: /var/unbound/access_lists.conf
                                      include: /var/unbound/host_entries.conf
                                      include: /var/unbound/dhcpleases_entries.conf
                                      include: /var/unbound/domainoverrides.conf
                                      forward-zone:
                                      	name: "."
                                      	forward-addr: 8.8.8.8
                                      	forward-addr: 8.8.4.4
                                      server:
                                       log-servfail: yes
                                       private-domain: "pfsense.mydomain.com"
                                      include: /var/unbound/remotecontrol.conf
                                      

                                      both "Register DHCP leases in the DNS Resolver" and "Register DHCP static mappings in the DNS Resolver" are enabled

                                      Lease time is default (24hrs, I think?)

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        jkv @hans109h
                                        last edited by Feb 20, 2021, 6:00 AM

                                        @hans109h

                                        I also had the “ Register DHCP leases in the DNS Resolver” option enabled when seeing this issue and disabling this option appears to have stopped the crash.

                                        M 1 Reply Last reply Feb 20, 2021, 6:02 AM Reply Quote 0
                                        • M
                                          maverick_slo @jkv
                                          last edited by Feb 20, 2021, 6:02 AM

                                          @jkv +1
                                          No packages at all...

                                          1 Reply Last reply Reply Quote 0
                                          41 out of 105
                                          • First post
                                            41/105
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.