To me at least it tells me that for sure states created due to FRR is omitted.

Edit: See https://forum.netgate.com/topic/196577.

@amithb we host hundreds of citrix sessions, and with the states low we are getting complaints about disconnects so we've change the settings back and will be looking to replace pfsense as our bgp router....just isn't working how we need it.

@Kevin-S-Pare - No worries. I think I got some ideas to try from the conversation here.

@michmoor
I actually have that enabled….i forced the gateway down but it still didn’t reset the states until it was actually down…

The "force gateway down feature" is a bit misleading and could probably use better wording and or further consideration. Forcing the gateway down doesn't trigger the same action as a link/packet issue hence why it doesn't kill states. If the gateway is forced down, it shouldn't be getting used so the states that already exist on it won't come back once they expire. If you want to immediately kill the states on top of preventing the gateway from being used, then you can force it down then separately kill states for that gateway from either Status > Gateways or Diagnostics > States.

FWIW it sounds like you may have TAC. They are a great resource and ultimately would have been able to explain/resolve both the state policy and gateway issue.

@michmoor brings up some good points about stateful filtering for routing. It does sound like TNSR would be better suited in your environment but that's not to say pfSense couldn't be made to work either. There have been times I've made the same decision to try an alternative product to then later on learn more about the issue and realize how it could have been solved. At the same time for better and worse the alternative brought its own challenges. Those kinds of things keep you humble.

If you have the Netgate 8200 or 8300 I forget which one you said, I would seriously reach out to Netgate sales and ask for TNSR. It’s close to Cisco syntax and it’s better suited for your task

Another option to think of and I'm not sure how well this would work is for each BGP peer, you have gateway monitoring enabled. Monitor IP can be whatever you want just different for each BGP peer.

There is an option when the gateway fails to kill states

https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#state-killing-gateway-failure

@Kevin-S-Pare said in [BGP Routing Issue: Traffic Still Preferring Default Route Despite

But reading about it, i'm not 100% sure its not still a stateful firewall?

Reply

Not sure what you are asking. Is an ASR1001 a firewall or router? Its a router. Routers are not stateful devices by nature. If you take advantage of the SDN side (the license is expensive) its a very robust platform.

If you want a cost-effective solution and still stick with netgate I know they offer TNSR. Ive been playing with it and its not bad. Granted I'm coming from an Arista/Juniper background so TNSR has some shortcomings that would prevent me from deploying in an enterprise but it does BGP. It can handle routes.

As an aside...I deployed Cumulus a few years ago and that turned me totally off on using OSS network gear. I made the exception with Netgate but man...I would never do that again.

The pfSense firewall just isn't meant to route at the edge using BGP. Minus the shortcoming you are seeing with FRR and pfsense holding onto states, I personally would not design any solution that requires tracking state and also doing bgp. Firewall behind the router.

You are very correct. it's just not working how I want it to work. I found a pretty good deal on a asr1009-20gb i'll pick up and try out instead.

But reading about it, i'm not 100% sure its not still a stateful firewall?

Reply

Not sure what you are asking. Is an ASR1001 a firewall or router? Its a router. Routers are not stateful devices by nature. If you take advantage of the SDN side (the license is expensive) its a very robust platform.

If you want a cost-effective solution and still stick with netgate I know they offer TNSR. Ive been playing with it and its not bad. Granted I'm coming from an Arista/Juniper background so TNSR has some shortcomings that would prevent me from deploying in an enterprise but it does BGP. It can handle routes.

As an aside...I deployed Cumulus a few years ago and that turned me totally off on using OSS network gear. I made the exception with Netgate but man...I would never do that again.

The pfSense firewall just isn't meant to route at the edge using BGP. Minus the shortcoming you are seeing with FRR and pfsense holding onto states, I personally would not design any solution that requires tracking state and also doing bgp. Firewall behind the router.

@Kevin-S-Pare

To be fair, i would never use a stateful device(firewall) to handle BGP routing to the internet.
You are even considering a Cisco ASR which is correct - use a router not a firewall.

I was actually considering picking up a ASR1001X-20G instead of running pfsense for my bgp peers.

But reading about it, i'm not 100% sure its not still a stateful firewall?

@Kevin-S-Pare Thanks for checking on this issue. I haven’t found a solution yet, so I’m currently managing with static routes as a workaround. Any guidance or suggestions to try out would be greatly appreciated.

I'll do some testing friday night and see how lowering the state timeout goes.