Wan failover doesn’t work and bigger problems
-
@idiotzoo said in Wan failover doesn’t work and bigger problems:
"State Killing on Gateway Failure"
Just recognised the problem with this - it doesn't reset the states when the gateway returns... I don't think this is the issue, as manually clearing the states doesn't resolve the issue, but is there no way to have the states reset when the gateways change?
One other thing I forgot to mention. When PFsense gets into this broken state on a restart, the web configurator doesn't work and needs to be restarted from the CLI.
-
Mmm, indeed it will not kill states that are in use when the tier one gateway comes back up. Doing so would be needlessly disruptive in many cases.
It's not usually an issue for most traffic since states timeout and new states are created on the tier 1 gateway seamlessly. However you might see issues for traffic that retains a state for long periods like VoIP for example.
One the group has failed back you should see that in Status > Gateways > Groups.
You can also check the firewall ruleset directly in /tmp/rules.debug. That file is updated with whatever gateway is current in the group.If the webgui cannot start I would expect something to be logged. A bad ruleset would not cause that but without something more to go on it's hard to say exactly what might.
If you see it again I would try to grab a status file before you reboot by browsing to <your_firewall_ip>/status.php directly. That should have enough info to show what happened.Steve
-
@stephenw10 That all makes sense... I'm not sure why clearing states doesn't seem to resolve the tier1 gateway returning.
My concern is this is a remote network that I support as a volunteer. My time is limited and I'm torn between spending time trying to chase down issues vs learning an alternative product. I'll try and make some time in the new year for some in depth testing. Recreating the issue on reboot is my greatest concern.
-
Mmm, I understand. For the WAN failover try to verify that new connections are at least using the primary WAN when it comes back up.
For the failing to start correctly case try to gather whatever info you can whilst it's in that state. -
@stephenw10 The site has just had a power outage (electrical testing) and the "no internet on reboot" problem showed itself.
As previously, I had to restart the web configurator.
By the time I got to the logs, I can't see anything obvious but haven't scoured it.
I have captured the status output both before and after re-running the setup wizard.
Anything specific I should be looking for? I'm happy to share things but not clear what needs redacting from the status output.
-
Also experienced a power outage. Both devices are back up and appear to be functioning as they should but OpenVPN and any external traffic is now being blocked. I am fairly new to the system so any suggestions on what I could check would be huge.
Thanks
-
@sethgko23 You could try running through the setup wizard again to see if that makes it work. That's what I have to do.
-
@idiotzoo What's the impact of that? Everything internally in the office is accessible it's just all our remote employees.
-
@sethgko23 Ah sounds like a different problem. I'm encountering no routing without having to re-run the wizard.
-
The first thing I would check in the status file is the routing table. Make sure there is a default route and it's valid.
Then I would check the state table file, make sure that are some states. If the ruleset is not loading there will be no open states.
Then the system log and the dmesg (message buffer).Steve
-
===group
===@stephenw10 Hi Steve,
I've captured the status in fault state and after running the wizard to get things going. It now looks like this issue can be reproduced - the site has just had some electrical work done and it seems to break after a power off.
Routing table looks fine. It's identical when things are working.
There are states
I can't see anything obvious in the system log, not sure what I'm looking for though.
Likewise the message buffer has no smoking gun... although there is this bit (in bold)
Configuring LAN interface...done.
Configuring GUESTWIFI interface...done.
Configuring BEELINE_WAN interface...done.
Configuring IPsec VTI interfaces...done.
Configuring CARP settings...done.
Syncing OpenVPN settings...
tun1: changing name to 'ovpns1'
pid 379 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Segmentation fault (core dumped)
Starting CRON... done.
Starting package lldpd...done.
Starting package Traffic Totals...done.
Starting package OpenVPN Client Export Utility...done.
Netgate pfSense Plus 21.05-RELEASE arm Tue Jun 01 16:52:45 EDT 2021
Bootup complete
pflog0: promiscuous mode enabledThere's no sea of errors. Everything looks fine but it doesn't work. When it's in the broken state I've tried making firewall and NAT rule changes to trigger a reload. The only thing that seems to make it work again is running the setup wizard.
-
@idiotzoo said in Wan failover doesn’t work and bigger problems:
pid 379 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Segmentation fault (core dumped)That's a pretty big smoking gun! If php crashes out at boot any number of the start up scripts may not have run.
I would still expect to see something configured incorrectly in the status file. Are you able to send it to me to review?
Just to be clear this only happens after a power outage and not ever after a normal reboot or shutdown/boot cycle?
Steve
-
@stephenw10 ha… so I highlighted the right thing.
I can certainly share the status files. Just didn’t want to post them publicly.
This first occurred when I initiated a reboot, so that should have been a polite shutdown. The rest of the reboots have been less so.
-
Ok, well if you can PM me a link that would work. Otherwise you could open a ticket and make it for my attention. I can review it when time allows.
Steve
-
@stephenw10 Thanks Steve, I've sent you a PM with the status details. Will be sure to update folks here with the outcome either way.
-
The logs show that was only up for ~2h before it stopped routing is that correct?
Three things jump out there:
You have SSH open to the world so your logs are full of random SSH login attempts and the resulting blocks.
Only the VoIP gateway group is actually failover. Most of your traffic is being policy routed via gateway groups that only have one gateway in them, is that intentional? Something you were doing for testing?
This is an SG-3100 running 21.05-REL. I should have spotted that before but I'd assumed you were on the latest version. You should upgrade to 21.05.2, there were specific fixes for the 3100 in 21.05.1 that addressed the php crash you're seeing and that could easily explain everything you're seeing.
https://docs.netgate.com/pfsense/en/latest/releases/21-05-1.htmlSteve
-
@stephenw10 Thanks Steve, appreciate you taking a look at it. In this case, I suspect that uptime is probably about right. Electrical safety tests meant things kept being switched off. The second power outage was unexpected when a circuit tripped out.
SSH being open to the world is certainly suboptimal, but it's been my only way in to make the system work after a power outage without a 2 hour drive. Ordinarily I would not do this.
The non-voip group is not using the failover as you spotted, basically because I couldn't make the failover work properly, well it would fail over but then stick and not switch back - or so it seemed, I have more testing to do there.
I'll update. The last time I ran an update everything died and I had a 2 hour drive to deal with it, hence I've resisted an update that I fear might kill the system or at least my access to it.
-
Hmm, well I would use a gateway directly rather than a group with one gateway in it.
21.05-REL on the 3100 has known issues that will cause you problems so I would update if at all possible.
Can you limit SSH to know IPs or to dyndns addresses perhaps?
Steve
-
@stephenw10 Hi Steve, it looks like issue 12004 is possibly what was causing the pain on boot issues. The significance of a PHP crash wasn't something I'd fully appreciated.
With 21.05.2 the system appears to start up happily.
It would be super helpful if a fix that affected platforms referenced the models in the release notes.
Thanks for your help. Just the WAN failover to do some more testing with.
-
@idiotzoo said in Wan failover doesn’t work and bigger problems:
helpful if a fix that affected platforms referenced the models in the release notes
Although the 3100 isn't directly mentioned in https://docs.netgate.com/pfsense/en/latest/releases/21-05-1.html it is one of the few 32 bit ARM models. (also the 1000).
