I am facing a challenging BGP routing issue in my pfSense setup within my lab environment, and I would greatly appreciate any insights from the community experts.
Setup Overview:
SFO Site (AS 65001) and LAX Site (AS 65002) connected via BGP.
Backbone network IPs:
SFO router WAN IP: 10.80.20.203
LAX router WAN IP: 10.80.21.141
Default route gateway: 10.80.20.1 (used as the backbone network's gateway for internet connectivity within the lab).
Additional AS (65003) for NSX-T networks in both SFO and LAX. This is working fine, no issue so far.
Goals:
Route traffic between SFO and LAX subnets using BGP to ensure direct communication, bypassing the default route (10.80.20.1).
Actions Taken:
Local Preference Adjustment: Set higher Local Preference (200) for direct BGP routes.
AS Path Prepending: Applied AS path prepending to deprioritize the path through 10.80.20.1.
MED Adjustment: Set MED to 200 for routes via 10.80.20.203 to make them less preferable.
Next Hop Self: Enabled to ensure proper route advertisement.
Firewall and NAT Rules: Verified that there are no conflicting rules affecting traffic flow.
Issue:
Despite these configurations, traffic between the SFO and LAX subnets is still preferring the default route (10.80.20.1) over the direct BGP-learned path. The BGP routing tables on both sides appear correct, and AS path prepending reflects properly. However, traceroute and packet captures show that traffic continues to take the path through the default gateway.
Routing Table Snapshots:
SFO BGP Routing Table:
K>* 0.0.0.0/0 [0/0] via 10.80.20.1, vmx0, 00:25:22
C>* 10.80.20.0/23 [0/1] is directly connected, vmx0, 00:25:22
B>* 172.17.11.0/24 [20/0] via 10.80.20.203, vmx0, weight 1, 00:25:17
LAX BGP Routing Table:
K>* 0.0.0.0/0 [0/0] via 10.80.20.1, vmx0, 00:25:29
C>* 10.80.20.0/23 [0/1] is directly connected, vmx0, 00:25:29
B>* 172.16.11.0/24 [20/0] via 10.80.21.141, vmx0, weight 1, 00:25:20
Troubleshooting Tried:
Increased Weight for BGP routes.
Ensured correct route-map configurations.
Confirmed that BGP attributes (MED, AS path) are applied.
Checked administrative distance settings.
Validated kernel routing tables against BGP tables.
Request for Help:
Are there any overlooked settings or best practices for BGP route selection in pfSense?
Could there be underlying factors or limitations with the way pfSense/FRR handles route preferences that we missed?
Any advice on further troubleshooting steps or configuration changes that could help prioritize the direct BGP routes over the default route?
Thank you in advance for any suggestions or guidance. Your expertise would be greatly appreciated!