Failover for Snapshot upgrade yesterday yields empty route table on reboot
-
I have a failover pair that I've been upgrading to 2.3 beta snapshots every few days. Unfortunately the reasons I've been upgrading still exist (failover states checked on firewall 1 cause the backup firewall to fail).
At any rate, I "temporarily disable carp on firewall 1 in order to fail it over to firewall 2 so I could perform a snapshot upgrade on firewall 1. The snapshot upgrade was a success and on reboot did not come back up. I went to see what the problem was and found that the routing table on reboot is completely empty and only returns "Routing Tables" when doing an netstat -nr, though the interfaces are configured.
Did I miss a memo or something? Also, I remember reading that there was an XMLRPC bug that caused the backup firewall in a pair to reboot. This still seems to be an issue, but ONLY if I have "Synchronize States" enabled.
-
Sounds like you're using limiters with pfsync, can't do that. Either disable the limiters or disable pfsync.
https://redmine.pfsense.org/issues/4310Should be no reason for no routes though. What routes are you expecting, just a default route, or?
-
Thank you for the link. Much appreciated! I wasn't aware of that bug! This isn't a "this is just the way it is" sort of thing is it? What I mean is, the target is to fix this. I have a network here that needs throttling/limiting BADLY. I feel like I'm tripping over one bug after another. I was on 2.1.1 and tried to go to 2.2.6 which has the IPSEC crash issue as well as XMLRPC sync issues with SNORT and CARP syncs.
When I say there are no routes, I mean literally no routes.(ha!) No default routes, no custom routes, no Local LAN routes. Literally it's.
#netstat -nr
Routing TablesObviously, I can't even ping on the local LAN. REALLY perplexing.
-
In that case it didn't even configure any network interfaces at all. It'd almost certainly have spit out some kind of error during boot in that case. My best guess on that would be unable to read the config at all (for lack of any better idea). What comes up at the console during boot? What does the console menu end up looking like?
-
OK You're right, my mistake, the interfaces aren't coming up at all. I'm able to configure the LAN interface, but any sort of action on the system (for example to restart the web configurator) gives the following action.
Fatal Error unable to create lock file: Bad File Descriptor.
There are no strange errors on reboot however I AM just dropped into a login prompt directly after / is mounted, as opposed to configuring WAN interface LAN interface and any OPT interfaces.
-
I wound up just rebuilding the system and upgrading it. We're SORT of back to normal.
I HAVE disabled the limiters. The actually limiter values are there but disabled, and the rulesets are set to NONE, however I'm still having the backup reboot even without limiters in ANY of the rulesets with pfsync enabled.
-
Nevermind. I think the lack of pfsync left a straggler rule on an interface. I've cleaned everything up, so far so good. I think I've run out of things to gripe about :). Looking forward to having the limiters working normally with HA.
-
Good to hear. pfsync and config sync are two different, unrelated components. You can leave the config sync enabled, and pfsync disabled, if you want to keep the limiters.
-
Yup, that I knew. Thanks a lot for all of your help CMB. I really appreciate your time. Looking forward to the continued updates!
Unfortunate, (and circumstancial) that the upgrade from 2.1.1 to 2.3 (so I can use IPSEC, HA, SNORT etc) had me tripping over so many bugs. I feel like I have a grasp on what happened and where, and really the only bug affecting the config is the pfsync and HA with limiters. The limiters are not detrimental as I actually have a separate pipeline I can funnel the traffic out of if I need to, so I can function. IPSEC and Snort were big ones for me, glad the IPSEC issue that plagued 2.2.x due to OS issues wasn't an issue in 2.3.
Again, my sincerest thanks!
-D