Status>Monitoring, no monitoring data logged while in WAN failover

marcg

Running 26.07.

I have two IPv4 interfaces in a gateway group configured for failover. From 5-7 AM this morning, my primary ISP went down. The secondary took over correctly: clients had v4 connectivity and I could see traffic flowing over the secondary interface in the Status>Dashboard Traffic Graphs widget. All good there.

However, Status>Monitoring shows no CPU activity logged during the outage (first pic). I rebooted the primary ISP modem and pfSense around 7:10 AM; data is logged correctly afterwards. Status>Monitoring also shows no traffic logged on the secondary interface during the outage (second pic) even though traffic was shown at those same times in the Status>Dashboard Traffic Graphs widget.

pfSense was running throughout the outage, passing traffic and logging to /var/log, e.g.,

[26.07-RELEASE][admin@pfSense.home.arpa]/var/log: grep 'Aug 18 05:[^0][0-9]' system.log.0 | wc -l
     128

Viper_Rus

@marcg
After updating to the latest version, monitoring is not working correctly on all my pfsense+ devices.

marcg

@Viper_Rus said in Status>Monitoring, no monitoring data logged while in WAN failover:

@marcg
After updating to the latest version, monitoring is not working correctly on all my pfsense+ devices.

No monitoring data is collected at all? My system is collecting CPU and gateway data most of the time, but definitely not during the WAN failover event mentioned above.

Viper_Rus

@marcg

Screenshot 2026-08-22 114512.png

It's showing nonsense. It's almost 12 o'clock now, I'm on the 22nd, and the statistics are for August 21st.

MaxK 0

@Viper_Rus My monitoring is not working either. I did a "Reset Data" to create a new file and I rebooted too. Neither worked. Also, there is no selection dropdown for the "Graph" when selecting the "Quality" Category.

Capture cPU.JPG

FireOdo

@Viper_Rus said in Status>Monitoring, no monitoring data logged while in WAN failover:

@marcg
After updating to the latest version, monitoring is not working correctly on all my pfsense+ devices.

... here too

MaxK 0

@stephenw10 Should this be tagged "Problems Installing or Upgrading pfSense software? If so, can you change it?

FireOdo

@Viper_Rus said in Status>Monitoring, no monitoring data logged while in WAN failover:

It's showing nonsense. It's almost 12 o'clock now, I'm on the 22nd, and the statistics are for August 21st.

Try, at the console, the option 11 (restart gui) wait a few minutes and see whats happend.
Also you can check if rrd is running: ps -ax | grep rrd

my 2 cents

netblues

There is definitely something wrong with status/monitoring.
It just stops logging.
This was first observed when running 26.07 rc.
Have deleted all past logs, reinstalled and restarted.
It was working for 6 days, but no more . Time on the graph is also stopped.
It seems it is related to a wan interface disable/enable.
Will check again on that hypothesis

Happens on all monitors, including the default.

FireOdo

@netblues said in Status>Monitoring, no monitoring data logged while in WAN failover:

There is definitely something wrong with status/monitoring.

It is - as far as I know. @stephenw10 is allready informed.

netblues

going into advaned, stop graph/enable graph restarts it

FireOdo

@netblues said in Status>Monitoring, no monitoring data logged while in WAN failover:

going into advaned, stop graph/enable graph restarts it

Thats a way too ...

luckman212

Hmm. Yep I just realized I'm having this problem too (26.07). Not sure if there's a redmine for this yet.

Last night, I noticed all my graph tabs looked like this:

When I checekd to see if updaterrd.sh was running with ps auxwwd | grep rrd - it wasn't.

I have a reaper daemon that cleans up stale (>24h with no writes) *.rrd files from /var/db/rrd. It's a simple cronjob that runs this script periodically:

#!/bin/sh
E_NOW=$(date +%s)
for r in /var/db/rrd/*.rrd
do
  E_RRD=$(stat -f "%m" $r)
  T_RRD=$(stat -f "%Sm" $r)
  E_DIFF=$(expr $E_NOW - $E_RRD)
  if [ $E_DIFF -gt 86400 ]; then
    rm $r
    /usr/bin/logger -t rrdclean "Purged orphan rrd db $r (timestamp: $T_RRD)"
  fi
done

Because it logs this activity, I was able to pinpoint the exact time the failure seems to have occurred. I think it was during a switch firmware upgrade where multiple LAN interfaces went down all at once, triggering some kind of hysteresis with the gateway monitoring actions.

I made a tiny PHP script that restarts it...

<?php
require_once("shaper.inc");
require_once("rrd.inc");
enable_rrd_graphing();

Running that got things moving again, but the script died again exactly 15 minutes later.

That's all I know for now, it looks like /etc/inc/rrd.inc code hasn't been touched in decades so might need some dusting off. Or maybe this is all a moot point now that Nexus is coming and the PHP interfaces are getting scrapped (?).

edit: when I checked, /var/run/updaterrd.sh.pid was not present. Looking at the monitoring graphs, they had been restarted at 8:30AM and failed at exactly 8:45AM:

screen 0.png

screen 1.png

A tail of my system.log showed only 1 entry at 8:45… check_reload_status is starting:

<13>1 2026-08-29T08:36:55.162653-04:00 r1.lan check_reload_status 676 - - Updating all dyndns
<13>1 2026-08-29T08:45:29.332042-04:00 r1.lan check_reload_status 72105 - - check_reload_status is starting.
<38>1 2026-08-29T08:47:34.842291-04:00 r1.lan sshd-session 51158 - - Accepted publickey for root from 192.168.20.40 port 57997 ssh2: ED25519 SHA256:xxx…

I'm running it in truss now with the -x flag on the updaterrd.sh shell script via the little helper below, will see if anything interesting gets logged.

#!/bin/sh

__kill() {
  kill -0 $pid 2>/dev/null || return 0
  echo "updaterrd.sh is running (pid $pid)"
  echo "stopping updaterrd.sh…"
  kill $pid
  sleep 10
}

if [ -e /var/run/updaterrd.sh.pid ]; then
  read pid < /var/run/updaterrd.sh.pid
  if [ -n "$pid" ]; then
    __kill $pid
  fi
fi

apid=$(pgrep -fx '/bin/sh /var/db/rrd/updaterrd.sh')
if [ -n "$apid" ]; then
  __kill $apid
fi

echo "starting updaterrd.sh in debug mode… (truss output -> /tmp/updaterrd.trace)"
truss -o /tmp/updaterrd.trace /bin/sh -x /var/db/rrd/updaterrd.sh

luckman212

Well, whatever's causing this definitely isn't "every 15 minutes" - been running for >2 hours now without failure. The hunt continues...

netblues

@luckman212 Has been running for 6days +.
Certainly not 15 minutes.

Look for interfaces flapping.

luckman212

@netblues interfaces flapping can and does happen in the real world... that shouldn't be enough to take down the RRD graphing system. This hasn't happened before and I've been running pfSense for 14+ years. Wanting to dig into this a bit more as it seems there's something here.

chpalmer

Running 2.9.0 CE here..

My lab machine is not always running so I will have to leave it running to see if it is also affected. Im sure though it is based on this thread..

netblues

@luckman212 said in Status>Monitoring, no monitoring data logged while in WAN failover:

@netblues interfaces flapping can and does happen in the real world... that shouldn't be enough to take down the RRD graphing system. This hasn't happened before and I've been running pfSense for 14+ years. Wanting to dig into this a bit more as it seems there's something here.

No one said that this is normal.
But it seems it could be the reason that it triggers the failure.

kikuyu

it seems to be related for me with the flapping on the igc3 interface.
This interface is for a secundary WAN and under 26.03.1 the monitoring was not down .

luckman212

I've been working on and testing this all day.

TL;DR seems to be a bit of a race condition and/or weirdness with the way lockf works on the latest FreeBSD builds. I don't know why, but I do know that the updaterrd.sh script that gets generated by /etc/inc/rrd.inc seems prone to deadlocking or failing to kill and respawn, leaving RRD monitoring effectively "dead" after interface flaps, config changes, or other unknown events.

This small patch adds some more aggressive killing (uses KILL sig if TERM fails to stop the daemon, uses a small retry loop to acquire the lock if the initial one fails, and switches the lockf call away from an exec so the result code can be captured and used to retry if needed. Also adds a bit of debug logging). Sample size of 1, YMMV, caveat emptor yada yada. But, it's fixed the problem for me.

diff/src over at GitHub gist. If you want to give it a go, add this to System Patches:

https://gist.githubusercontent.com/luckman212/9685be13e67ad59b0225528bae99a389/raw/0f35eed98ee0eb5bba6a7473c9320b73cd905b5a/rrd.inc.patch

After applying, restart the RRD backend with this 1-liner from console:

php -r 'require_once("shaper.inc"); require_once("rrd.inc"); enable_rrd_graphing();'

BONUS: I also found a bug in check_reload_status where it can enter a CPU-spinning event loop after FastCGI EOF... I was hitting this one too (I think it might be new for 26.07 as well, but there are some old redmines e.g. #14891 and the positively ancient #2555, so it could be I'm just lucky.) With some AI assistance I have a patch and have been running that too for 48+ hrs… so far so good. I will submit the findings, the patch and the compiled x64 drop in binary replacement shortly for anyone who wants to test that too.

edit: see Submitting a PR for check_reload_status