BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments
-
I am facing a challenging BGP routing issue in my pfSense setup within my lab environment, and I would greatly appreciate any insights from the community experts.
Setup Overview:
SFO Site (AS 65001) and LAX Site (AS 65002) connected via BGP.
Backbone network IPs:
SFO router WAN IP: 10.80.20.203
LAX router WAN IP: 10.80.21.141
Default route gateway: 10.80.20.1 (used as the backbone network's gateway for internet connectivity within the lab).
Additional AS (65003) for NSX-T networks in both SFO and LAX. This is working fine, no issue so far.Goals:
Route traffic between SFO and LAX subnets using BGP to ensure direct communication, bypassing the default route (10.80.20.1).
Actions Taken:
Local Preference Adjustment: Set higher Local Preference (200) for direct BGP routes.
AS Path Prepending: Applied AS path prepending to deprioritize the path through 10.80.20.1.
MED Adjustment: Set MED to 200 for routes via 10.80.20.203 to make them less preferable.
Next Hop Self: Enabled to ensure proper route advertisement.Firewall and NAT Rules: Verified that there are no conflicting rules affecting traffic flow.
Issue:
Despite these configurations, traffic between the SFO and LAX subnets is still preferring the default route (10.80.20.1) over the direct BGP-learned path. The BGP routing tables on both sides appear correct, and AS path prepending reflects properly. However, traceroute and packet captures show that traffic continues to take the path through the default gateway.
Routing Table Snapshots:
SFO BGP Routing Table:K>* 0.0.0.0/0 [0/0] via 10.80.20.1, vmx0, 00:25:22
C>* 10.80.20.0/23 [0/1] is directly connected, vmx0, 00:25:22
B>* 172.17.11.0/24 [20/0] via 10.80.20.203, vmx0, weight 1, 00:25:17LAX BGP Routing Table:
K>* 0.0.0.0/0 [0/0] via 10.80.20.1, vmx0, 00:25:29
C>* 10.80.20.0/23 [0/1] is directly connected, vmx0, 00:25:29
B>* 172.16.11.0/24 [20/0] via 10.80.21.141, vmx0, weight 1, 00:25:20Troubleshooting Tried:
Increased Weight for BGP routes.
Ensured correct route-map configurations.
Confirmed that BGP attributes (MED, AS path) are applied.
Checked administrative distance settings.
Validated kernel routing tables against BGP tables.Request for Help:
Are there any overlooked settings or best practices for BGP route selection in pfSense?
Could there be underlying factors or limitations with the way pfSense/FRR handles route preferences that we missed?Any advice on further troubleshooting steps or configuration changes that could help prioritize the direct BGP routes over the default route?
Thank you in advance for any suggestions or guidance. Your expertise would be greatly appreciated!
-
@amithb said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
te these configurations, traffic between the SFO and LAX subnets is still preferring the default route (10.80.20.1) over the direct BGP-learned path. The BGP routing tables on both sides appear correct, and AS path prepending reflects properly. However, traceroute and packet captures show that traffic continues to take the path through the default gateway.
Routing Table Snapshots:
Are you doing any policy based routing? Setting a specific gateway in the firewall rules? Thats the only thing i can imagine that would bypass the route table.
-
@michmoor said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
Are you doing any policy based routing? Setting a specific gateway in the firewall rules? Thats the only thing i can imagine that would bypass the route table.
There is no policy based routing or specific gateway in firewall rule. Only one default gateway present on both site routers. I would say it is just a basic BGP config. I just followed the example configuration from the netgate docs - https://docs.netgate.com/pfsense/en/latest/packages/frr/bgp/example.html
-
@amithb One thing i've noticed is that when I gracefully shutdown a bgp peer, incoming traffic moved over to the other peer. but if there were enteries in the state table using the shutdown peer, traffic would still go out that peer. i'm just looking into TCP Established and its default is 86400 or 24 hours. i've turned mine down to 90 seconds and will do some testing there with that.
-
Known issue.
https://redmine.pfsense.org/issues/14630
https://redmine.pfsense.org/issues/14633#note-2Supposedly, script support is needed in FRR to fix
@marcosm Any fix incoming for this? -
@michmoor said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
Known issue.
https://redmine.pfsense.org/issues/14630
https://redmine.pfsense.org/issues/14633#note-2Supposedly, script support is needed in FRR to fix
@marcosm Any fix incoming for this?Shouldn't a low state time help fix this issue too? or are the states just kinda glued in there.
I even tried marking the gateway as down but until I actually unplugged the cable and made it down, then it killed the states....not very graceful lol....
im trying to not spend money on some cisco isr for my bgp, but this is just making my case unfortunately....
-
To be fair, i would never use a stateful device(firewall) to handle BGP routing to the internet.
You are even considering a Cisco ASR which is correct - use a router not a firewall. -
This post is deleted! -
@Kevin-S-Pare Thanks for checking on this issue. I haven’t found a solution yet, so I’m currently managing with static routes as a workaround. Any guidance or suggestions to try out would be greatly appreciated.
-
@amithb said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
@Kevin-S-Pare Thanks for checking on this issue. I haven’t found a solution yet, so I’m currently managing with static routes as a workaround. Any guidance or suggestions to try out would be greatly appreciated.
I'll do some testing friday night and see how lowering the state timeout goes.
-
@michmoor said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
To be fair, i would never use a stateful device(firewall) to handle BGP routing to the internet.
You are even considering a Cisco ASR which is correct - use a router not a firewall.I was actually considering picking up a ASR1001X-20G instead of running pfsense for my bgp peers.
But reading about it, i'm not 100% sure its not still a stateful firewall?
-
@amithb we host hundreds of citrix sessions, and with the states low we are getting complaints about disconnects so we've change the settings back and will be looking to replace pfsense as our bgp router....just isn't working how we need it.
-
@Kevin-S-Pare said in [BGP Routing Issue: Traffic Still Preferring Default Route Despite
But reading about it, i'm not 100% sure its not still a stateful firewall?
Reply
Not sure what you are asking. Is an ASR1001 a firewall or router? Its a router. Routers are not stateful devices by nature. If you take advantage of the SDN side (the license is expensive) its a very robust platform.
If you want a cost-effective solution and still stick with netgate I know they offer TNSR. Ive been playing with it and its not bad. Granted I'm coming from an Arista/Juniper background so TNSR has some shortcomings that would prevent me from deploying in an enterprise but it does BGP. It can handle routes.
As an aside...I deployed Cumulus a few years ago and that turned me totally off on using OSS network gear. I made the exception with Netgate but man...I would never do that again.
The pfSense firewall just isn't meant to route at the edge using BGP. Minus the shortcoming you are seeing with FRR and pfsense holding onto states, I personally would not design any solution that requires tracking state and also doing bgp. Firewall behind the router.
-
@michmoor said in BGP Routing Issue: Traffic Still Preferring Default Route Despite Prepending and MED Adjustments:
@Kevin-S-Pare said in [BGP Routing Issue: Traffic Still Preferring Default Route Despite
But reading about it, i'm not 100% sure its not still a stateful firewall?
Reply
Not sure what you are asking. Is an ASR1001 a firewall or router? Its a router. Routers are not stateful devices by nature. If you take advantage of the SDN side (the license is expensive) its a very robust platform.
If you want a cost-effective solution and still stick with netgate I know they offer TNSR. Ive been playing with it and its not bad. Granted I'm coming from an Arista/Juniper background so TNSR has some shortcomings that would prevent me from deploying in an enterprise but it does BGP. It can handle routes.
As an aside...I deployed Cumulus a few years ago and that turned me totally off on using OSS network gear. I made the exception with Netgate but man...I would never do that again.
The pfSense firewall just isn't meant to route at the edge using BGP. Minus the shortcoming you are seeing with FRR and pfsense holding onto states, I personally would not design any solution that requires tracking state and also doing bgp. Firewall behind the router.
You are very correct. it's just not working how I want it to work. I found a pretty good deal on a asr1009-20gb i'll pick up and try out instead.
-
Another option to think of and I'm not sure how well this would work is for each BGP peer, you have gateway monitoring enabled. Monitor IP can be whatever you want just different for each BGP peer.
There is an option when the gateway fails to kill states
https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#state-killing-gateway-failure
-
@michmoor
I actually have that enabled….i forced the gateway down but it still didn’t reset the states until it was actually down… -
@Kevin-S-Pare
Ok yeah that sucks…migrate… -
@michmoor yup! Ordering the Cisco today
-
@Kevin-S-Pare
Just frustrating.
This is a similar situation I ran into with Cumulus. I’m all for open source software and do want to support but there are just situations I find myself in where something basic just doesn’t work. whether it’s an IPsec bug or dynamic routing. It’s just frustrating so I understand where you are coming from. -
@michmoor we’ve done some amazing stuff with Netgate so I can’t complain….they are doing great things but they have their limits and their place…