Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    possible bug: updaterrd.sh left not running after rapid successive rc.newwanipv6 / rc.newwanip events (WAN save with DHCPv6), all RRD graphs stop silently

    Scheduled Pinned Locked Moved IPv6
    2 Posts 2 Posters 155 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      therealmaxidude
      last edited by therealmaxidude

      Can anyone validate this is a correct observation ?

      Project: pfSense (CE)
      Tracker: possible bug
      Category: Monitoring (RRD graphs) / Gateway Monitoring
      Version: 2.9.0-RELEASE
      Platform: amd64, VM on Proxmox VE (virtio NICs, Open vSwitch bridges)

      Subject:
      updaterrd.sh left not running after rapid successive rc.newwanipv6 /
      rc.newwanip events (WAN save with DHCPv6), all RRD graphs stop silently

      DESCRIPTION

      After saving the WAN interface on a system with DHCP (IPv4) + DHCPv6 with
      prefix delegation on WAN, the RRD collector (/var/db/rrd/updaterrd.sh) can
      end up not running at all. All RRD data (quality, system, traffic, etc.)
      stops being recorded. Nothing is logged about it, and the firewall
      otherwise keeps working normally.

      The collector only came back about 6 hours later, when a routine DHCPv6
      Renew triggered rc.newwanipv6, which ran "Creating rrd update script" one
      more time.

      In the log, a single WAN save produced, within about 30 seconds:

      • 4x "Creating rrd update script" (3 of them within 5 seconds)
      • 4x rc.newwanipv6 runs (REQUEST, RELEASE, RELEASE, REQUEST)
      • 4x "Restarting/Starting all packages"
      • 14x "dpinger: exiting on signal 15"
      • several "[Gateway Monitoring] status socket dpinger: not found" errors

      The running collector is started as:
      /usr/bin/lockf -s -k -t 0 /var/run/updaterrd.sh.pid /bin/sh /var/db/rrd/updaterrd.sh

      Suspected cause: the RRD script is regenerated and restarted several times
      in parallel, and the new instance uses lockf with "-t 0" (fail immediately
      if the lock is held). When one invocation kills the running collector while
      another still holds the lock, the instance that should survive exits
      because the lock is busy. The final result is that no collector is running.
      (This is inferred from the log sequence and the process arguments. I did
      not trace it in the code.)

      STEPS TO REPRODUCE

      1. WAN: IPv4 DHCP + IPv6 DHCPv6 with prefix delegation (/57, prefix hint
        sent, "Do not wait for a RA" off), LAN: Track Interface.
        Upstream: ISP CPE (Telenet CV8560E) acting as DHCPv6 server.
      2. Gateway monitoring enabled on WAN_DHCP and WAN_DHCP6.
      3. System > Routing > Gateways: edit WAN_DHCP6, save + apply.
      4. About 15 seconds later: Interfaces > WAN, save + apply.
      5. Check: ps auxww | grep [u]pdaterrd
        -> no updaterrd.sh process
        Status > Monitoring -> no new data from this point on.

      This is not reproduced on every save. It is timing dependent.

      EXPECTED RESULT

      Exactly one updaterrd.sh instance is running after any interface or gateway
      change, and RRD graphs continue without a gap.

      ACTUAL RESULT

      No updaterrd.sh running. RRD gap from 10:28 (last data shown about 10:35 due
      to heartbeat) until 16:45, when a DHCPv6 Renew triggered rc.newwanipv6 ->
      "Creating rrd update script" and the collector came back.

      rrdtool fetch on WAN_DHCP-quality.rrd (last 24h, transitions only):
      10:35 GAP
      16:50 ok

      Process after recovery:
      root 75913 ... 16:45 /usr/bin/lockf -s -k -t 0 /var/run/updaterrd.sh.pid /bin/sh /var/db/rrd/updaterrd.sh
      root 76233 ... 16:45 /bin/sh /var/db/rrd/updaterrd.sh

      No clock or NTP issues: there are no "illegal attempt to update" errors and
      no ntpd step messages in system.log.

      SECONDARY OBSERVATION (possibly a separate issue)

      The restart storm is amplified by dhcp6c cycling several times during one
      WAN apply:

      10:28:40 dhcp6c: Bypassing address release because of -n flag (clean stop, -n honoured)
      10:28:43 dhcp6c: Solicit/Request -> NA + PD success
      10:28:44 rtsold: INFO [DHCP Client] RTSOLD Lock in place - sending SIGHUP to dhcp6c
      10:28:44 rtsold: <call_script> write to child failed: Broken pipe
      10:28:49 dhcp6c: restarting
      10:28:49 dhcp6c: Start address release / Sending Release (release IS sent here)
      10:28:51 dhcp6c: Sending Release (again)
      10:28:53 dhcp6c: Solicit/Request -> NA + PD success (XID mismatch logged)

      So a single WAN save leads to: Request -> SIGHUP from rtsold -> Release ->
      Release -> Request. Each step fires rc.newwanipv6, which restarts packages,
      dpinger and the RRD script. Also, the SIGHUP path sends a DHCPv6 Release
      even though dhcp6c honours -n on a normal stop.

      Related: on every DHCPv6 Renew, pfSense reports a WAN IP change between the
      SLAAC address and the DHCPv6 IA_NA /128 and restarts all packages:
      16:45:36 NOTICE pfSense package system has detected an IP change or dynamic
      WAN reconnection - 2a02:...:a89c:d526 -> 2a02:...::883a - Restarting packages.

      WORKAROUND

      • Status > Monitoring > Settings (wrench) > Save, which restarts the collector.
      • Or a cron safety net:
        pgrep -qf updaterrd.sh || /usr/local/bin/php -r 'require_once("config.inc"); require_once("rrd.inc"); enable_rrd_graphing();'

      ENVIRONMENT DETAILS

      • pfSense 2.9.0-RELEASE, PHP 8.5.12 (updated 2026-10-21/22 via pfSense-upgrade + pkg upgrade)
      • Packages: Suricata (inline on WAN), pfBlockerNG, OpenVPN, lldpd, Service Watchdog, Cron
      • Gateway monitoring: WAN_DHCP (IPv4), WAN_DHCP6 (monitor IP 2606:4700:4700::1111)
      • Local logging enabled + remote syslog

      attachment logs: redmine-updaterrd-appendix-log-masked.txt

      1 Reply Last reply
      Reply Quote 0
      • A
        artenpie
        last edited by

        Same as this post ? If so, luckman212's patch is in that post. Look for v3. Worked for me.

        1 Reply Last reply
        Reply Quote 0

        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

        With your input, this post could be even better 💗

        Register Login
        • First post
          Last post
        Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.
        Privacy Policy · Cookie Policy