possible bug: updaterrd.sh left not running after rapid successive rc.newwanipv6 / rc.newwanip events (WAN save with DHCPv6), all RRD graphs stop silently
-
Can anyone validate this is a correct observation ?
Project: pfSense (CE)
Tracker: possible bug
Category: Monitoring (RRD graphs) / Gateway Monitoring
Version: 2.9.0-RELEASE
Platform: amd64, VM on Proxmox VE (virtio NICs, Open vSwitch bridges)Subject:
updaterrd.sh left not running after rapid successive rc.newwanipv6 /
rc.newwanip events (WAN save with DHCPv6), all RRD graphs stop silentlyDESCRIPTION
After saving the WAN interface on a system with DHCP (IPv4) + DHCPv6 with
prefix delegation on WAN, the RRD collector (/var/db/rrd/updaterrd.sh) can
end up not running at all. All RRD data (quality, system, traffic, etc.)
stops being recorded. Nothing is logged about it, and the firewall
otherwise keeps working normally.The collector only came back about 6 hours later, when a routine DHCPv6
Renew triggered rc.newwanipv6, which ran "Creating rrd update script" one
more time.In the log, a single WAN save produced, within about 30 seconds:
- 4x "Creating rrd update script" (3 of them within 5 seconds)
- 4x rc.newwanipv6 runs (REQUEST, RELEASE, RELEASE, REQUEST)
- 4x "Restarting/Starting all packages"
- 14x "dpinger: exiting on signal 15"
- several "[Gateway Monitoring] status socket dpinger: not found" errors
The running collector is started as:
/usr/bin/lockf -s -k -t 0 /var/run/updaterrd.sh.pid /bin/sh /var/db/rrd/updaterrd.shSuspected cause: the RRD script is regenerated and restarted several times
in parallel, and the new instance uses lockf with "-t 0" (fail immediately
if the lock is held). When one invocation kills the running collector while
another still holds the lock, the instance that should survive exits
because the lock is busy. The final result is that no collector is running.
(This is inferred from the log sequence and the process arguments. I did
not trace it in the code.)STEPS TO REPRODUCE
- WAN: IPv4 DHCP + IPv6 DHCPv6 with prefix delegation (/57, prefix hint
sent, "Do not wait for a RA" off), LAN: Track Interface.
Upstream: ISP CPE (Telenet CV8560E) acting as DHCPv6 server. - Gateway monitoring enabled on WAN_DHCP and WAN_DHCP6.
- System > Routing > Gateways: edit WAN_DHCP6, save + apply.
- About 15 seconds later: Interfaces > WAN, save + apply.
- Check: ps auxww | grep [u]pdaterrd
-> no updaterrd.sh process
Status > Monitoring -> no new data from this point on.
This is not reproduced on every save. It is timing dependent.
EXPECTED RESULT
Exactly one updaterrd.sh instance is running after any interface or gateway
change, and RRD graphs continue without a gap.ACTUAL RESULT
No updaterrd.sh running. RRD gap from 10:28 (last data shown about 10:35 due
to heartbeat) until 16:45, when a DHCPv6 Renew triggered rc.newwanipv6 ->
"Creating rrd update script" and the collector came back.rrdtool fetch on WAN_DHCP-quality.rrd (last 24h, transitions only):
10:35 GAP
16:50 okProcess after recovery:
root 75913 ... 16:45 /usr/bin/lockf -s -k -t 0 /var/run/updaterrd.sh.pid /bin/sh /var/db/rrd/updaterrd.sh
root 76233 ... 16:45 /bin/sh /var/db/rrd/updaterrd.shNo clock or NTP issues: there are no "illegal attempt to update" errors and
no ntpd step messages in system.log.SECONDARY OBSERVATION (possibly a separate issue)
The restart storm is amplified by dhcp6c cycling several times during one
WAN apply:10:28:40 dhcp6c: Bypassing address release because of -n flag (clean stop, -n honoured)
10:28:43 dhcp6c: Solicit/Request -> NA + PD success
10:28:44 rtsold: INFO [DHCP Client] RTSOLD Lock in place - sending SIGHUP to dhcp6c
10:28:44 rtsold: <call_script> write to child failed: Broken pipe
10:28:49 dhcp6c: restarting
10:28:49 dhcp6c: Start address release / Sending Release (release IS sent here)
10:28:51 dhcp6c: Sending Release (again)
10:28:53 dhcp6c: Solicit/Request -> NA + PD success (XID mismatch logged)So a single WAN save leads to: Request -> SIGHUP from rtsold -> Release ->
Release -> Request. Each step fires rc.newwanipv6, which restarts packages,
dpinger and the RRD script. Also, the SIGHUP path sends a DHCPv6 Release
even though dhcp6c honours -n on a normal stop.Related: on every DHCPv6 Renew, pfSense reports a WAN IP change between the
SLAAC address and the DHCPv6 IA_NA /128 and restarts all packages:
16:45:36 NOTICE pfSense package system has detected an IP change or dynamic
WAN reconnection - 2a02:...:a89c:d526 -> 2a02:...::883a - Restarting packages.WORKAROUND
- Status > Monitoring > Settings (wrench) > Save, which restarts the collector.
- Or a cron safety net:
pgrep -qf updaterrd.sh || /usr/local/bin/php -r 'require_once("config.inc"); require_once("rrd.inc"); enable_rrd_graphing();'
ENVIRONMENT DETAILS
- pfSense 2.9.0-RELEASE, PHP 8.5.12 (updated 2026-10-21/22 via pfSense-upgrade + pkg upgrade)
- Packages: Suricata (inline on WAN), pfBlockerNG, OpenVPN, lldpd, Service Watchdog, Cron
- Gateway monitoring: WAN_DHCP (IPv4), WAN_DHCP6 (monitor IP 2606:4700:4700::1111)
- Local logging enabled + remote syslog
attachment logs: redmine-updaterrd-appendix-log-masked.txt
-
Same as this post ? If so, luckman212's patch is in that post. Look for v3. Worked for me.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register LoginPrivacy Policy · Cookie Policy