Unbound stopped after WAN interface goes down/up
-
Hello,
Running pfSense 2.3-RELEASE (amd64) (at home, on DSL).
I've got a recurring issue where unbound is in a stopped state after the WAN interface changes state. Typically this occurs if the power goes off (and kills the vDSL modem, but pfSense box is on a UPS so remains up) - so the WAN i/f goes down for a short while, then comes back up.
When this occurs, unbound shows as stopped on the dashboard, and Internet access is "off" as there's no DNS resolution. Manually starting unbound from the dashboard solves the issue (bit of a pain for others if I'm not around !).
Here's the syslog general output:
Syslog: Apr 25 19:36:00 kernel igb0: link state changed to DOWN Apr 25 19:36:00 check_reload_status Linkup starting igb0 Apr 25 19:36:01 php-fpm 40206 /rc.linkup: DEVD Ethernet detached event for wan Apr 25 19:36:11 check_reload_status Linkup starting igb0 Apr 25 19:36:11 kernel igb0: link state changed to UP Apr 25 19:36:12 php-fpm 52457 /rc.linkup: DEVD Ethernet attached event for wan Apr 25 19:36:12 php-fpm 52457 /rc.linkup: HOTPLUG: Configuring interface wan Apr 25 19:36:15 check_reload_status updating dyndns WAN_DHCP Apr 25 19:36:15 check_reload_status Restarting ipsec tunnels Apr 25 19:36:15 check_reload_status Restarting OpenVPN tunnels/interfaces Apr 25 19:36:15 check_reload_status Reloading filter Apr 25 19:36:16 xinetd 16396 Starting reconfiguration Apr 25 19:36:16 xinetd 16396 Swapping defaults Apr 25 19:36:16 xinetd 16396 readjusting service 6969-udp Apr 25 19:36:16 xinetd 16396 Reconfigured: new=0 old=1 dropped=0 (services) Apr 25 19:37:29 check_reload_status rc.newwanip starting igb0 Apr 25 19:37:29 php-fpm 52457 /rc.linkup: ROUTING: setting default route to a.b.c.d Apr 25 19:37:29 php-fpm 52457 /rc.linkup: The command '/sbin/route change -inet default 'a.b.c.d'' returned exit code '1', the output was 'route: writing to routing socket: No such process route: writing to routing socket: Network is unreachable change net default: gateway a.b.c.d fib 0: Network is unreachable' Apr 25 19:37:29 check_reload_status Restarting ipsec tunnels Apr 25 19:37:30 php-fpm 76698 /rc.newwanip: rc.newwanip: Info: starting on igb0. Apr 25 19:37:30 php-fpm 76698 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: WAN[wan]) (real interface: igb0). Apr 25 19:37:30 php-fpm 76698 /rc.newwanip: rc.newwanip: Failed to update wan IP, restarting... Apr 25 19:37:30 check_reload_status Configuring interface wan Apr 25 19:37:32 check_reload_status rc.newwanip starting igb0 Apr 25 19:37:33 check_reload_status updating dyndns wan Apr 25 19:37:33 php-fpm 5558 /rc.newwanip: rc.newwanip: Info: starting on igb0. Apr 25 19:37:33 php-fpm 5558 /rc.newwanip: rc.newwanip: on (IP address: a.b.c.d) (interface: WAN[wan]) (real interface: igb0). Apr 25 19:37:33 xinetd 16396 Starting reconfiguration Apr 25 19:37:33 xinetd 16396 Swapping defaults Apr 25 19:37:33 xinetd 16396 readjusting service 6969-udp Apr 25 19:37:33 xinetd 16396 Reconfigured: new=0 old=1 dropped=0 (services) Apr 25 19:37:39 php-fpm 5558 /rc.newwanip: ROUTING: setting default route to a.b.c.d Apr 25 19:37:42 php-fpm 5558 /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1461609462] unbound[19327:0] error: bind: address already in use [1461609462] unbound[19327:0] fatal error: could not open ports' Apr 25 19:37:43 php-fpm 5558 /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Apr 25 19:37:43 php-fpm 5558 /rc.newwanip: Creating rrd update script Apr 25 19:37:45 php-fpm 5558 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - - Restarting packages. Apr 25 19:37:45 check_reload_status Starting packages Apr 25 19:37:46 php-fpm 5558 /rc.start_packages: Restarting/Starting all packages. Apr 25 19:37:46 php-fpm 5558 [pfBlockerNG] Starting cron process.And the DNS resolver log from the same time:
Apr 25 19:37:29 unbound 16872:0 info: service stopped (unbound 1.5.5). <bunch of="" stat="" lines...="">Apr 25 19:37:54 unbound 5837:0 notice: init module 0: validator Apr 25 19:37:55 unbound 5837:0 notice: init module 1: iterator Apr 25 19:37:55 unbound 5837:0 info: start of service (unbound 1.5.5). Apr 25 19:37:55 unbound 5837:0 info: service stopped (unbound 1.5.5). Apr 25 19:37:55 unbound 5837:0 info: server stats for thread 0: 46 queries, 0 answers from cache, 46 recursions, 0 prefetch Apr 25 19:37:55 unbound 5837:0 info: server stats for thread 0: requestlist max 15 avg 10.1087 exceeded 0 jostled 0 <bunch more="" stat="" lines...=""><manual start="" of="" unbound="" from="" the="" dahboard="">Apr 25 19:59:47 unbound 89546:0 notice: init module 0: validator Apr 25 19:59:47 unbound 89546:0 notice: init module 1: iterator Apr 25 19:59:47 unbound 89546:0 info: start of service (unbound 1.5.5).</manual></bunch></bunch>Unbound appears to start, then immediately stop.
The interesting line from the general log is:```
The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1461609462] unbound[19327:0] error: bind: address already in use [1461609462] unbound[19327:0] fatal error: could not open ports'Although the timing of this doesn't quite fit in with the DNS resolver log. I am running pfBlockerNG, but disabled this and triggered the issue (switched the modem off/on) - and the same issue occurred, so it doesn't appear to be pfBlockerNG related. Any ideas ? Thanks. -
Still no idea why unbound stops when the WAN interface flaps, but I've installed the 'Service Watchdog' package, so hopefully that will auto start unbound in the future.
-
I have the same problem. Today i updated to 2.3_1 and installed watchdog. Hopefully it solves the problem. It also didn't stop every time wan goes down. Every night there is a 24h disconnect and some times the dns resolver didn't start. I only have that problem since updating to 2.3.
-
Not sure why people think 2.3_1 made any changes/fixes to anything - in wasn't anything special.. It was an update to ntpd nothing more..
Yes service watchdog can be helpful in restarting services that stop for whatever reason.
-
Not sure why people think 2.3_1 made any changes/fixes to anything
Agreed - the problem I was having occurred on 2.2.6 / 2.3 / 2.3_1, so for me it's not related to 2.3_1.
The watchdog is working well - no DNS issues since I installed it (and the WAN interface has been down/up a few times since).
-
Just to report the same issue on 2.3.2-RELEASE (amd64). I have dual wan configuration, and I've used DNS Forwarder until few days ago, when I moved my DNS to unbound. DNS Forwarder was working fine all the time.
Today, I had some problems with both WAN connections, and few hours later I've realized that hosts behind the pfSense do not resolve.
After restarting unbound service, everything works fine.