I'm trying to get BGP working between Metal LB on my Kubernetes cluster and pfSense (2.7.2)/FRR (2.0.2_1).
My aim is to use the 192.168.254.0/24 network for my Kubernetes load balancer, advertised by metallb via BGP to pfsense, so I can access those addresses from my LAN. The Kubernetes cluster has nodes in the 10.10.10.0/24 network on vlan 254 and my local 192.168.1.0/24 network is on VLAN 10.
However I've clearly done something wrong because my traffic never reaches my ingress objects. In fact, traceroute to one of the ingress addresses just seems to exhaust the TTL. This was working under VyOS, so I suspect it's a pfsense configuration issue (combined with the fact I have no idea what I'm doing)
Can anyone spot any obvious config issues with the below? I hope this rambling post makes some sense.
2d92e7f7-d3f3-4a41-819a-4e5d84764341-Screenshot_20250608_124627-1.png
pfSense FRR-BGP configuration
frr defaults traditional
hostname ahostname
password redacted123
ip nht resolve-via-default
service integrated-vtysh-config
!
ip router-id 10.10.10.1
!
ip route 192.168.254.0/24 10.10.10.10
ip route 192.168.254.0/24 10.10.10.11
ip route 192.168.254.0/24 10.10.10.12
ip route 192.168.254.0/24 10.10.10.13
!
router bgp 64512
bgp router-id 10.10.10.1
bgp graceful-shutdown
no bgp network import-check
no bgp ebgp-requires-policy
neighbor metallb peer-group
neighbor metallb remote-as 64511
neighbor metallb update-source 10.10.10.1
neighbor 10.10.10.10 peer-group metallb
neighbor 10.10.10.10 remote-as 64511
neighbor 10.10.10.10 description Kube Master
neighbor 10.10.10.11 peer-group metallb
neighbor 10.10.10.11 remote-as 64511
neighbor 10.10.10.12 peer-group metallb
neighbor 10.10.10.12 remote-as 64511
neighbor 10.10.10.13 remote-as 64511
!
address-family ipv4 unicast
network 192.168.254.0/24
neighbor 10.10.10.10 activate
neighbor 10.10.10.11 activate
neighbor 10.10.10.12 activate
neighbor 10.10.10.13 activate
no neighbor metallb send-community
no neighbor 10.10.10.10 send-community
no neighbor 10.10.10.11 send-community
no neighbor 10.10.10.12 send-community
no neighbor 10.10.10.13 send-community
exit-address-family
!
!
line vty
!
Metal LB configuration
---
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: metallb-bgp-peer
namespace: metallb-system
spec:
myASN: 64511
peerASN: 64512
peerAddress: 10.10.10.1
enableGracefulRestart: true
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: address-pool-bgp
namespace: metallb-system
spec:
addresses:
- 192.168.254.0/24
autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: metal-lb-bgp-adv
namespace: metallb-system
spec:
ipAddressPools:
- address-pool-bgp
aggregationLength: 24
localPref: 100
an example ingress object and some diagnostic output:
NAME CLASS HOSTS ADDRESS PORTS AGE
kuard traefik test.mydomain.com 192.168.254.1 80, 443 133d
$ kubectl --namespace metallb-system get bgpadvertisements.metallb.io
NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS PEERS
metal-lb-bgp-adv ["address-pool-bgp"]
$ kubectl --namespace metallb-system get bgppeers.metallb.io
NAME ADDRESS ASN BFD PROFILE MULTI HOPS
metallb-bgp-peer 10.10.10.1 64512
Traceroute output:
$ traceroute 192.168.254.1
traceroute to 192.168.254.1 (192.168.254.1), 30 hops max, 60 byte packets
1 _gateway (192.168.1.1) 0.170 ms 0.092 ms 0.075 ms
2 rpi-cm4-tp2-04.domain.com (10.10.10.13) 0.311 ms 0.357 ms 0.335 ms
3 10.10.10.1 (10.10.10.1) 0.351 ms 0.461 ms 0.440 ms
4 rpi-cm4-tp2-04.domain.com (10.10.10.13) 0.609 ms 0.658 ms 0.706 ms
5 10.10.10.1 (10.10.10.1) 0.607 ms 0.650 ms 0.604 ms
6 * * *
7 10.10.10.1 (10.10.10.1) 0.770 ms 0.764 ms 0.717 ms
8 * * *
9 10.10.10.1 (10.10.10.1) 1.250 ms 1.245 ms 1.235 ms
10 * * *
11 10.10.10.1 (10.10.10.1) 1.574 ms 1.535 ms 1.565 ms
12 * * *
13 10.10.10.1 (10.10.10.1) 1.240 ms 1.250 ms 0.988 ms
14 * * *
15 10.10.10.1 (10.10.10.1) 0.994 ms 0.890 ms 0.929 ms
16 * * *
17 10.10.10.1 (10.10.10.1) 0.949 ms 0.794 ms 0.970 ms
18 * * *
19 10.10.10.1 (10.10.10.1) 1.061 ms 2.817 ms 2.779 ms
20 * * *
21 10.10.10.1 (10.10.10.1) 3.028 ms 3.001 ms 2.953 ms
22 * * *
23 10.10.10.1 (10.10.10.1) 2.363 ms 2.352 ms 2.514 ms
24 * * *
25 10.10.10.1 (10.10.10.1) 2.662 ms 2.645 ms 2.631 ms
26 * * *
27 10.10.10.1 (10.10.10.1) 2.689 ms 2.700 ms 2.668 ms
28 * * *
29 10.10.10.1 (10.10.10.1) 1.747 ms 1.381 ms 1.128 ms
30 * * *