Status>Monitoring, no monitoring data logged while in WAN failover

kikuyu

@stephenw10
My pfsense box has 4 RJ45 ports. The port igc3: <Intel(R) Ethernet Controller I226-V> is connected to a netgear LM 1200 LTE for a WAN backup. Sometimes the LM1200 is losing his LTE but the up/down of the link has never affected the monitoring in the version 26.03-1

marcg

@stephenw10 , No PPPoE in my setup. See the first post in the thread. Thanks!

Edit: Interfaces are all I226-V (2.5 GbE, igc driver). The one carrying the backup WAN has multiple VLANs, including one dedicated to the backup modem. No VLANs on the interface with the primary WAN. @luckman212

FireOdo

@luckman212 said in Status>Monitoring, no monitoring data logged while in WAN failover:

I've been working on and testing this all day.

Thank you! Patch applied (2.9.0 CE) - observing now!

EDIT1: RRD is not stopping anymore (02.09.26)
EDIT2: Still everything fine - 11 entrys in the syslog (I guess for debugging)

Regards,
FireOdo

stephenw10

@luckman212 said in Status>Monitoring, no monitoring data logged while in WAN failover:

a single WAN cable pull and reconnect, ️12 DEVD attach/detach events were fired.

All for the same device? Sub-interfaces on the same link perhaps?

I agree that seems like at least 6x too many!

luckman212

@stephenw10 This was a WAN iface pull, no other ifs or vlans on it. I haven't tested the same on other hardware yet but I do remember it being pretty noisy on my 6100 too.

dennypage

@luckman212 Well done sir!

luckman212

anyone besides @FireOdo, @0x44 or @kikuyu try the patch? Would love to get some more testing. It's been working fine for me.

chpalmer

@stephenw10 said in Status>Monitoring, no monitoring data logged while in WAN failover:

Mmm, interesting, I've been trying to replicate it here and was failing for some time. But I have now found that I can only replicate it on a system that has a PPPoE WAN enabled and is able to connect.

Is anyone seeing this on a system without PPPoE?

Happening on my Watchguard M470 box with no PPPoe.

Ive just applied Luckman's patch and I am monitoring as well.

netblues

Easy to replicate.

Just disabling a dhcp wan interface from interface is enough to bring down rrd

After applying the patch and retesting, issue isn't occurring anymore

Thanks

0x44

@luckman212 It's working fine. Thanks a lot.

0x44

@luckman212 Have you filed a bug report in redmine?

luckman212

Not yet, I've been away traveling this weekend (big holiday weekend here in the US) and haven't had time to finish the work I'm doing with fixing check_reload_status which is separate but related. I wanted to wrap it all up together. I expect to finish that tonight / tomorrow.

luckman212

Just want to update anyone following this thread:

I've been working on the (important) companion patch of check_reload_status for the last week. I'm up to version 10 now. It's been stable for a couple of days, and has survived all sorts of failure modes:

WANs going offline (soft)
Ethernet attach/detach events (hard), including "storms" of rapid up/down sequences
A Python torture test suite that throws various forms of nonsense at the socket: garbage/truncated commands, split headers, oversized packets, aborted connections, floods of duplicate commands, etc.

It's looking good but I want to test it for a little longer before submitting the patch. If anyone would like a drop-in compiled binary for amd64 let me know- I would love some additional testers!

stephenw10

@luckman212 Sure I can run it on a few things.

luckman212

Thanks @stephenw10 that would be awesome. I'm packaging it up now and will post details in a few hrs.

luckman212

@stephenw10 I put a downloadable check_reload_status-0.0.18.pkg together and wrapped it all up with the source and a bunch of notes at https://github.com/luckman212/check-reload-status.

Would love if you or anyone else could please test!

chpalmer

@luckman212 said in Status>Monitoring, no monitoring data logged while in WAN failover:

Would love if you or anyone else could please test!

"nice: Badly formed number."

expected?

I am on 2.9.0CE

luckman212

@chpalmer no, that's not expected. can you tell me where/when you saw that message? during install? or in the logs somewhere?

chpalmer

@luckman212 said in Status>Monitoring, no monitoring data logged while in WAN failover:

@chpalmer during install? or in the logs somewhere?

When copy paste each of these commands..
pkill -9 check_reload_status
nice -n20 /usr/local/sbin/check_reload_status

On both my primary and lab machines now.

2.9.0 primary
26.07+ lab

Installing both via ssh using putty.

luckman212

@chpalmer Ah, that pesky tcsh or whatever it is. Try:

/bin/pkill -9 check_reload_status
/usr/bin/nice -n20 /usr/local/sbin/check_reload_status

edit: thanks, I've updated the docs and the pkg-message to use the full pathnames so it works in either shell.