Unbound restarting on v6 IA_NA and IA_PD updates when address and prefixes are unchanged
-
Running 26.03.1.
My router gets its GUA and delegated prefixes via DHCPv6 from my ISP (ATT fiber, BGW320). All are stable -- haven't changed in years -- even though they are "dynamic." However, unbound restarts whenever the leases are renewed, even though the addresses/prefixes haven't changed. Note that LAN-side DNS registrations of KEA leases is disabled; this is a WAN-side issue.
There are three delegated prefixes. Because ATT delegates individual /64s, each renewal cycle involves a GUA and three prefixes. ATT sets the T1 and T2 timers to both be 1800 -- don't know why -- which results (?) in each renewal cycle containing a Rebind and a Renew for each object. First snippet below shows a typical cycle for updating all leases, filtered from the dhcpd.log.
Each renew and rebind causes a restart of unbound, even though the addresses/prefixes don't change. Second snippet below is filtered from the resolver.log for the same cycle. Every 30 minutes, unbound is effectively down for 30s, which causes problems elsewhere in the network.
In this example, there are 12 restarts of unbound even though there are only 8 rebinds and renews. Haven't been able to figure out the extra 4 restarts. They don't correlate with v4 DHCP activity on the WAN. In fact, WAN DHCPv4 leases are stably renewed every 5 minutes with no apparent unbound restarts.
The WAN interface has a DHCP6 configuration file override to manage the multiple prefix requests, modeled after this thread. The config file is the final snippet below. I've looked at
/var/etc/dhcp6c_wan_script.shand the scripts that it calls, in particular/etc/rc.newwanipv6which looks like it should (?) detect whether addresses/prefixes have changed.Thanks in advance for suggestions and solutions.
Jun 2 13:24:59 pfSense dhcp6c[75478]: Sending Renew Jun 2 13:24:59 pfSense dhcp6c[75478]: Sending Rebind Jun 2 13:24:59 pfSense dhcp6c[75478]: update an IA: NA-0 Jun 2 13:24:59 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:00 pfSense dhcp6c[75478]: update an IA: NA-0 Jun 2 13:25:00 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:11 pfSense dhcp6c[75478]: Sending Renew Jun 2 13:25:11 pfSense dhcp6c[75478]: Sending Rebind Jun 2 13:25:11 pfSense dhcp6c[75478]: update an IA: PD-2 Jun 2 13:25:11 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:12 pfSense dhcp6c[75478]: update an IA: PD-2 Jun 2 13:25:12 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:19 pfSense dhcp6c[75478]: Sending Renew Jun 2 13:25:19 pfSense dhcp6c[75478]: Sending Rebind Jun 2 13:25:19 pfSense dhcp6c[75478]: update an IA: PD-1 Jun 2 13:25:19 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:20 pfSense dhcp6c[75478]: update an IA: PD-1 Jun 2 13:25:20 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:27 pfSense dhcp6c[75478]: Sending Renew Jun 2 13:25:27 pfSense dhcp6c[75478]: Sending Rebind Jun 2 13:25:27 pfSense dhcp6c[75478]: update an IA: PD-0 Jun 2 13:25:27 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.sh Jun 2 13:25:28 pfSense dhcp6c[75478]: update an IA: PD-0 Jun 2 13:25:28 pfSense dhcp6c[75478]: executes /var/etc/dhcp6c_wan_script.shJun 2 13:25:02 pfSense unbound[20394]: [20394:0] info: start of service (unbound 1.25.1). Jun 2 13:25:04 pfSense unbound[76603]: [76603:0] info: start of service (unbound 1.25.1). Jun 2 13:25:06 pfSense unbound[32360]: [32360:0] info: start of service (unbound 1.25.1). Jun 2 13:25:15 pfSense unbound[27410]: [27410:0] info: start of service (unbound 1.25.1). Jun 2 13:25:16 pfSense unbound[85797]: [85797:0] info: start of service (unbound 1.25.1). Jun 2 13:25:18 pfSense unbound[44322]: [44322:0] info: start of service (unbound 1.25.1). Jun 2 13:25:22 pfSense unbound[33386]: [33386:0] info: start of service (unbound 1.25.1). Jun 2 13:25:24 pfSense unbound[93801]: [93801:0] info: start of service (unbound 1.25.1). Jun 2 13:25:26 pfSense unbound[56072]: [56072:0] info: start of service (unbound 1.25.1). Jun 2 13:25:31 pfSense unbound[44315]: [44315:0] info: start of service (unbound 1.25.1). Jun 2 13:25:32 pfSense unbound[2376]: [2376:0] info: start of service (unbound 1.25.1). Jun 2 13:25:34 pfSense unbound[58694]: [58694:0] info: start of service (unbound 1.25.1).interface igc0 { send ia-na 0; send ia-pd 0; send ia-pd 1; send ia-pd 2; request domain-name-servers; request domain-name; script "/var/etc/dhcp6c_wan_script.sh"; }; id-assoc na 0 { }; id-assoc pd 0 { prefix-interface igc1.15 { sla-id 0; sla-len 0; }; }; id-assoc pd 1 { prefix-interface igc1.20 { sla-id 0; sla-len 0; }; }; id-assoc pd 2 { prefix-interface igc1.40 { sla-id 0; sla-len 0; }; }; -
@marcg said in Unbound restarting on v6 IA_NA and IA_PD updates when address and prefixes are unchanged:
in particular /etc/rc.newwanipv6 which looks like it should (?) detect whether addresses/prefixes have changed
You didn't show the system log at the same moment. Nothing in there ?
If the reason was a DHCPv6 'renew' and 'nothing' changed, then according to here /etc/rc.newwanipv6 simply return, doing nothing.
Add a copy of line 190 (the log line) just before the 'return;' ? so you can check that is is - or is not 'rc.newwanipv6' that restarts unbound (and probably other services).
Modify it somewhat so it shows interesting info in the (system ?) log.@marcg said in Unbound restarting on v6 IA_NA and IA_PD updates when address and prefixes are unchanged:
ATT sets the T1 and T2 timers to both be 1800 -- don't know why --

== a dhcp6c renew every 5 minutes or 300 seconds .
This includes prefix info (I presume).
I get it, that feels strange. But as you said, GUAs, prefix are always dynamic even when the seem to be static. Mine are the same for the last 18+ months now. That makes us the lucky ones. There are pretty sick ISP out there that change them all the time. -
@Gertjan, thanks for the tip to look at system.log.
TL;DR: rc.newwanipv6 doesn't appear to be the issue. The LAN-side DHCPv6 server (Kea) and gateway groups may be involved.
I added lines to rc.newwanipv6 to check whether (a) the Reason code was being correctly passed and (b) whether the script was restarting unbound.
55a56,57 > logger(LOG_NOTICE, localize_text("rc.newwanipv6 invoked with reason: %s", $reason)); > 249a252,253 > logger(LOG_NOTICE, localize_text("rc.newwanipv6 restarting services, address_replaced is %s", $address_replaced ? "True" : "False")); >The system.log results below for one of the update cycles show that the Reason code is passed and the script is not restarting unbound (no messages from the second logger addition above).
[26.03.1-RELEASE][admin@pfSense.home.arpa]/etc: grep '10:2.*rc.newwanipv6' /var/log/system.log Jun 3 10:25:46 pfSense php-fpm[15732]: NOTICE rc.newwanipv6 invoked with reason: RENEW Jun 3 10:25:47 pfSense php-fpm[15732]: NOTICE rc.newwanipv6 invoked with reason: REBIND Jun 3 10:25:58 pfSense php-fpm[76079]: NOTICE rc.newwanipv6 invoked with reason: RENEW Jun 3 10:25:59 pfSense php-fpm[76079]: NOTICE rc.newwanipv6 invoked with reason: REBIND Jun 3 10:26:06 pfSense php-fpm[619]: NOTICE rc.newwanipv6 invoked with reason: RENEW Jun 3 10:26:07 pfSense php-fpm[619]: NOTICE rc.newwanipv6 invoked with reason: REBIND Jun 3 10:26:14 pfSense php-fpm[95728]: NOTICE rc.newwanipv6 invoked with reason: RENEW Jun 3 10:26:15 pfSense php-fpm[95728]: NOTICE rc.newwanipv6 invoked with reason: REBINDYou didn't show the system log at the same moment. Nothing in there ?
system.log does show the following at approximately the same time as each update cycle with its unbound restarts: four repetitions of DHCPv6 complaints, followed by a filter reload and state killing on igc1.95 (a different interface). Don't know if it's related, but four IA_NA+IA_PDs are being updated, although there are eight DHCP updates in total since each IA is both renewed and rebound.
Maybe unrelated to the unbound issue, but the timing matches that of the DHCPv6 WAN-side renewals. If it is related, that's bizarre for several reasons.
- DHCPv6 service is enabled on three LAN-side interfaces only, not nine as shown in each repetition in the log complaints. There are ULA VIPS on each of those interfaces as well as GUA prefixes. Maybe that gets you to six (3x2) but not to nine.
- No DNS registration of Kea leases
- igc1.95 is an interface for an IPv4-only Tier 2 backup gateway. It's in a gateway group with the Tier 1 dual-stack gateway that we've been discussing so far. The Tier 2 gateway is not going up and down in sync with the DHCP renewals on the WAN interface to the Tier 1 gateway.
- The router's WAN interface on igc1.95 is a static IPv4.
- No LAN-side DHCP ranges are changing.
Jun 3 10:25:48 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:48 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:48 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:50 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:50 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:50 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:51 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:51 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:51 pfSense php-fpm[15732]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:25:52 pfSense check_reload_status[670]: Reloading filter Jun 3 10:25:53 pfSense php-fpm[618]: NOTICE [Gateway Monitoring] killed all states for gateways with interface igc1.95 due to gateway recovery for IPv4_GW_GROUP Jun 3 10:26:00 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:00 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:00 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:02 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:02 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:02 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:04 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:04 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:04 pfSense php-fpm[76079]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:05 pfSense check_reload_status[670]: Reloading filter Jun 3 10:26:06 pfSense php-fpm[81917]: NOTICE [Gateway Monitoring] killed all states for gateways with interface igc1.95 due to gateway recovery for IPv4_GW_GROUP Jun 3 10:26:09 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:09 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:09 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:10 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:10 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:10 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:12 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:12 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:12 pfSense php-fpm[619]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:13 pfSense check_reload_status[670]: Reloading filter Jun 3 10:26:14 pfSense php-fpm[66525]: NOTICE [Gateway Monitoring] killed all states for gateways with interface igc1.95 due to gateway recovery for IPv4_GW_GROUP Jun 3 10:26:16 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:16 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:16 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:18 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:18 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:18 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:19 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:19 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:19 pfSense php-fpm[95728]: WARNING [DHCP Server] The specified range lies outside of the current subnet. Skipping DHCP6 entry. Jun 3 10:26:20 pfSense check_reload_status[670]: Reloading filter Jun 3 10:26:21 pfSense php-fpm[15732]: NOTICE [Gateway Monitoring] killed all states for gateways with interface igc1.95 due to gateway recovery for IPv4_GW_GROUP