WebGUI Crash on Reboot, CARP Issues, and WireGuard Package Hangs
-
I wasn't entirely sure where to post this, I have 3 odd issues going on with my backup firewall at a site and am starting to think they are kinda related, so not sure 3 separate posts makes sense.
Firstly, this backup firewall will randomly and erroneously grab Master for a single subnet/VIP at a time, there is no consistency as to which one it is though.
When this happens, I can still see VRRP advertisements reaching this firewall, so I don't think it's a layer 2 breakdown. Restarting CARP does not fix the issue, rebooting the firewall almost always does until it happens again (which can take days or weeks).
Notably, this firewall also can't be rebooted from the webGUI anymore, when doing so as soon as I click Reboot, the GUI hangs for about 5 minutes and then gives ma 5xx error.
If I login to the firewall via console and then issue a reboot there, it sometimes works but gets stuck on Stopping Package Wireguard sometimes for up to an hour.
The CARP issue is my biggest concern, as it grabbed Master on our WAN interface once which caused a lot of routing issues for pretty critical assets (I mean, that's why we have redundancy anyway).
Both units are Netgate 1541's with the added CPIC card and dual SSDs in a mirror. And I am running 25.07.1 w/ pfSense Plus.
Do these symptoms line up with anything that could all sort of be the same root cause? I'm starting to think these units maybe just need replacing, they were ROCK SOLID for like 5 years but I've had numerous issues recently (more than these 3 but with the others I found the root causes)
If needed I'll contact Netgate, but I always like to post here first in case it can be resolved in the community.
-
@planedrop said in WebGUI Crash on Reboot, CARP Issues, and WireGuard Package Hangs:
When this happens, I can still see VRRP advertisements reaching this firewall, so I don't think it's a layer 2 breakdown.
How are you testing that?
Are they arriving at the expected rate?
What do you see logged on the secondary when it becomes master on that interface?
-
@stephenw10 Just doing a pcap to see if the advertisements are showing up, they appear to be coming in at the expected intervals. To be clear I am not seeing flapping either, the secondary will take over Master, and until it reboots it'll just keep Master status.
Once it reboots, it'll go into Backup state and remain there until the next time this issue happens.
I had to reboot this secondary today and it also hung on the Wireguard issue, had to force kill it in order to get it back up (gave it a couple hours).
The log event I'm seeing when the secondary took over Master status is:
Carp master event
carp: 19@ interface : Backup > Master (master timed out)
HA cluster member hase resumed CARP state MasterNotably, the secondary does have a lot of "Carp backup event" and "carp: demoted by 240 to 240 (interface down)" events. These seemed to happen when the GUI was hanging when I was trying to reboot it.
That log makes me think it's not receiving the advertisements, but it's odd to have it only happen on a single VLAN and not the entire interface.
These firewalls are on the same switch to, both of which allow all tagged VLANs for these particular ports. Switches aren't showing any storm control events or anything like that, but to be fair they are Unifi switches so not always the most reliable, so maybe there is something breaking down that I'm not aware of at Layer 2.
-
They still show as tagged correctly in the VLAN? Do you see the secondary sending CARP advertisements at the slightly longer interval?
Mmm, just one VLAN on a NIC is odd indeed. Seems to rule out any sort of hardware issue.... unless maybe it's hardware VLAN tagging. Does it see other traffic arrive on that VLAN when it fails?
Which NICs are the VLANs on? Always on the same NIC type?
