BGP configuration help
-
I'm trying to get BGP working between Metal LB on my Kubernetes cluster and pfSense (2.7.2)/FRR (2.0.2_1).
My aim is to use the 192.168.254.0/24 network for my Kubernetes load balancer, advertised by metallb via BGP to pfsense, so I can access those addresses from my LAN. The Kubernetes cluster has nodes in the 10.10.10.0/24 network on vlan 254 and my local 192.168.1.0/24 network is on VLAN 10.However I've clearly done something wrong because my traffic never reaches my ingress objects. In fact, traceroute to one of the ingress addresses just seems to exhaust the TTL. This was working under VyOS, so I suspect it's a pfsense configuration issue (combined with the fact I have no idea what I'm doing)
Can anyone spot any obvious config issues with the below? I hope this rambling post makes some sense.
pfSense FRR-BGP configuration
frr defaults traditional hostname ahostname password redacted123 ip nht resolve-via-default service integrated-vtysh-config ! ip router-id 10.10.10.1 ! ip route 192.168.254.0/24 10.10.10.10 ip route 192.168.254.0/24 10.10.10.11 ip route 192.168.254.0/24 10.10.10.12 ip route 192.168.254.0/24 10.10.10.13 ! router bgp 64512 bgp router-id 10.10.10.1 bgp graceful-shutdown no bgp network import-check no bgp ebgp-requires-policy neighbor metallb peer-group neighbor metallb remote-as 64511 neighbor metallb update-source 10.10.10.1 neighbor 10.10.10.10 peer-group metallb neighbor 10.10.10.10 remote-as 64511 neighbor 10.10.10.10 description Kube Master neighbor 10.10.10.11 peer-group metallb neighbor 10.10.10.11 remote-as 64511 neighbor 10.10.10.12 peer-group metallb neighbor 10.10.10.12 remote-as 64511 neighbor 10.10.10.13 remote-as 64511 ! address-family ipv4 unicast network 192.168.254.0/24 neighbor 10.10.10.10 activate neighbor 10.10.10.11 activate neighbor 10.10.10.12 activate neighbor 10.10.10.13 activate no neighbor metallb send-community no neighbor 10.10.10.10 send-community no neighbor 10.10.10.11 send-community no neighbor 10.10.10.12 send-community no neighbor 10.10.10.13 send-community exit-address-family ! ! line vty !
Metal LB configuration
--- apiVersion: metallb.io/v1beta2 kind: BGPPeer metadata: name: metallb-bgp-peer namespace: metallb-system spec: myASN: 64511 peerASN: 64512 peerAddress: 10.10.10.1 enableGracefulRestart: true --- apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: address-pool-bgp namespace: metallb-system spec: addresses: - 192.168.254.0/24 autoAssign: true --- apiVersion: metallb.io/v1beta1 kind: BGPAdvertisement metadata: name: metal-lb-bgp-adv namespace: metallb-system spec: ipAddressPools: - address-pool-bgp aggregationLength: 24 localPref: 100
an example ingress object and some diagnostic output:
NAME CLASS HOSTS ADDRESS PORTS AGE kuard traefik test.mydomain.com 192.168.254.1 80, 443 133d
$ kubectl --namespace metallb-system get bgpadvertisements.metallb.io NAME IPADDRESSPOOLS IPADDRESSPOOL SELECTORS PEERS metal-lb-bgp-adv ["address-pool-bgp"] $ kubectl --namespace metallb-system get bgppeers.metallb.io NAME ADDRESS ASN BFD PROFILE MULTI HOPS metallb-bgp-peer 10.10.10.1 64512
Traceroute output:
$ traceroute 192.168.254.1 traceroute to 192.168.254.1 (192.168.254.1), 30 hops max, 60 byte packets 1 _gateway (192.168.1.1) 0.170 ms 0.092 ms 0.075 ms 2 rpi-cm4-tp2-04.domain.com (10.10.10.13) 0.311 ms 0.357 ms 0.335 ms 3 10.10.10.1 (10.10.10.1) 0.351 ms 0.461 ms 0.440 ms 4 rpi-cm4-tp2-04.domain.com (10.10.10.13) 0.609 ms 0.658 ms 0.706 ms 5 10.10.10.1 (10.10.10.1) 0.607 ms 0.650 ms 0.604 ms 6 * * * 7 10.10.10.1 (10.10.10.1) 0.770 ms 0.764 ms 0.717 ms 8 * * * 9 10.10.10.1 (10.10.10.1) 1.250 ms 1.245 ms 1.235 ms 10 * * * 11 10.10.10.1 (10.10.10.1) 1.574 ms 1.535 ms 1.565 ms 12 * * * 13 10.10.10.1 (10.10.10.1) 1.240 ms 1.250 ms 0.988 ms 14 * * * 15 10.10.10.1 (10.10.10.1) 0.994 ms 0.890 ms 0.929 ms 16 * * * 17 10.10.10.1 (10.10.10.1) 0.949 ms 0.794 ms 0.970 ms 18 * * * 19 10.10.10.1 (10.10.10.1) 1.061 ms 2.817 ms 2.779 ms 20 * * * 21 10.10.10.1 (10.10.10.1) 3.028 ms 3.001 ms 2.953 ms 22 * * * 23 10.10.10.1 (10.10.10.1) 2.363 ms 2.352 ms 2.514 ms 24 * * * 25 10.10.10.1 (10.10.10.1) 2.662 ms 2.645 ms 2.631 ms 26 * * * 27 10.10.10.1 (10.10.10.1) 2.689 ms 2.700 ms 2.668 ms 28 * * * 29 10.10.10.1 (10.10.10.1) 1.747 ms 1.381 ms 1.128 ms 30 * * *