24.03 FRR has flapping BGP neighbors
-
IPSec has some unique features with how it's filtered. What sort of IPSec tunnel(s) are you using, route or policy mode? How is the 'IPsec Filter Mode' set in IPSec Advanced Settings?
Depending on those you can have outbound states on the assigned interface and reply traffic on 'ipsec' and hence can be tripped up by the interface bound states.
-
@stephenw10
Howdy StephenI am using route based IPsec - VTI.
IPsec Filter Mode set to 'Filter IPsec Tunnel, Transport and VTI on IPsec tab'So are you saying there are two interfaces involved here? VTI and encap0 ?
If so i can logically see it being asymmetrical but i would like you to confirm. Its def a nuance point specifically with IPsec that should be documented i think. -
Yup pretty much exactly as you described it in the bug: https://redmine.pfsense.org/issues/15430.
The same issue that prevents NAT working on VTI interfaces. [https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474
-
@stephenw10
Makes sense.
Ive always wondered and you have more familiarity with whats being used out there but should i instead use the following?
I understand the limitation here is that i can no longer use IPsec mobile but if this is only going to be a IPsec gateway does it matter?
What are customers using out in the field?And if i make the above change does that mean the asymmetrical problem is solved?
-
It's not just mobile IPSec that is blocked it's also anything in policy mode (not VTI). If you're only using VTI mode IPSec then set that and it removes most if the limitations. Including this with state binding.
-
@stephenw10
i think thats reasonable for me then.
So this turns pfsense into a strickly VTI/Routed IPsec gateway.
The remote end can do whatever it wants technically, right? (policy or routed).The only downside is if you have a large amount of IPsec tunnels. They all get their own interface and firewall rules but would that be a GUI limitation displaying all of that?
-
@michmoor
Indeed the other end can do whatever it wants. However, I've found that having routed on one and policy on the other is prone to config mistakes so I would not normally recommend that. -
Ha! Yup if you're looking for a bad time and confusing diagnosis try mixing route and policy based IPSec.
-
M mcury referenced this topic on
-
i'm having the same issue, 23.09 vti ipsec tunnels worked great with frr/bgp, now they keep flapping. if i want to go back to 23.09, where would i get that image? or, what is the fix if there is one?
jim
-
If you're running ZFS you can just roll back the Boot Environment.
The Net Installer can install a number of versions including 23.09.1.
But you should first just try switching the State Interface Binding back to floating:
https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#config-advanced-firewall-state-policy -
@stephenw10 thanks! i rolled back and everything working great...This is the first time i've had to do that.
jim -
@stephenw10 so, how will i know its ok to upgrade in the future? will they have a release note about frr fix possibly?
jim
-
Well that's why I suggested switching the state binding back to floating. If that allows BGP to come up correctly in 24.03 then the fix here is to add floating rules for the VTI tunnels (if you have those).
The state binding changed in 24.03 to make it more secure and that isn't likely to be changed back. The underlying issue with VTI interfaces is being looked at but until then you need floating state binding rules for it. -
@stephenw10 ok, i'll try a test on a non production firewall :) when you say add floating rules, what exactly do you mean?
jim
-
-
@stephenw10 the flapping only seems to happen when both ends are on 24.03, i'll keep testing with my dev firewalls.
jim
-
@michmoor hi mich, can you give more detail on what rules you created to allow bgp across the interfaces?
thanks
jim
