FRR/Kubernetes/Metallb LAN traffic anomalies
-
Hey all, new to dealing with FRR, BGP, and pfSense in general, and am hoping I can get some help.
I am running a Kubernetes lab using MetalLB to publish load-balanced IPs and FRR to publish routes for that network. My traffic from the internet via NAT rule behaves as expected, but there is some odd behavior when it comes to traffic trying to hit my main application load balancer from my internal network. Here is a diagram to explain how everything is laid out:
LAN 10.250.0.0/16
BGP Route to 10.254.254.0/24
FRR ASN: 64512
MetalLB ASN: 64513Most http operations seem to work correctly, but some particular operations do not.
A packet capture taken while performing a test from an internal client shows a ton of TCP retransmits against the load-balancer IP. This capture was taken while uploading a document to an application which ingests PDF documents (paperless-ngx,) and the client yields a very generic sort of 'transfer failed' message.
Pinging the host returns a bunch of ICMP redirect messaging, which I've read doesn't seem to be a big deal. I have disabled firewall rules for traffic leaving on the same interface, as recommended in the document related to asymmetric routing.
Below are screenshots of the relevant configuration options and status information, please let me know if I can provide anything else:
Firewall WAN Rules (no LAN rules created for this, disabled firewall rules for LAN interface)
FRR Global Options
FRR Route Maps
FRR Route Map Details
FRR BGP Router Configuration
FRR BGP Router Advanced Configuration
FRR BGP Neighbor List
FRR BGP Neighbor example configuration
BGP Routes
BGP table version is 34, local router ID is 10.250.0.1, vrf id 0 Default local pref 100, local AS 64512 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 10.254.254.0/24 0.0.0.0 0 32768 i *= 10.254.254.1/32 10.250.100.3 0 64513 i *= 10.250.100.2 0 64513 i *= 10.250.100.5 0 64513 i *= 10.250.100.4 0 64513 i *> 10.250.100.1 0 64513 i *> 10.254.254.3/32 10.250.100.1 0 64513 i Displayed 3 routes and 7 total pathsBGP Summary
IPv4 Unicast Summary: BGP router identifier 10.250.0.1, local AS number 64512 vrf-id 0 BGP table version 34 RIB entries 4, using 768 bytes of memory Peers 5, using 71 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt 10.250.100.1 4 64513 42675 42691 0 0 0 01w5d06h 2 3 10.250.100.2 4 64513 42663 42694 0 0 0 01w5d06h 1 3 10.250.100.3 4 64513 42651 42675 0 0 0 01w5d06h 1 3 10.250.100.4 4 64513 42668 42701 0 0 0 01w5d06h 1 3 10.250.100.5 4 64513 42665 42697 0 0 0 01w5d06h 1 3 Total number of neighbors 5BGP Neighbors
BGP neighbor is 10.250.100.1, remote AS 64513, local AS 64512, external link BGP version 4, remote router ID 10.250.100.1, local router ID 10.250.0.1 BGP state = Established, up for 01w5d06h Last read 00:00:23, Last write 00:00:21 Hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast Route refresh: advertised Address Family IPv4 Unicast: advertised and received Address Family IPv6 Unicast: received Hostname Capability: advertised (name: fj-rt.frijole.lol,domain name: n/a) not received Graceful Restart Capability: advertised Graceful restart information: Local GR Mode: Helper* Remote GR Mode: Disable R bit: False Timers: Configured Restart Time(sec): 120 Received Restart Time(sec): 0 Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 5 5 Notifications: 6 0 Updates: 18 16 Keepalives: 42662 42654 Route Refresh: 0 0 Capability: 0 0 Total: 42691 42675 Minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast Update group 5, subgroup 12 Packet Queue length 0 Community attribute sent to this neighbor(large) 2 accepted prefixes Connections established 5; dropped 4 Last reset 01w5d06h, No AFI/SAFI activated for peer Local host: 10.250.0.1, Local port: 179 Foreign host: 10.250.100.1, Foreign port: 46183 Nexthop: 10.250.0.1 Nexthop global: fdf6:5825:2efe:728b:6662:66ff:fe21:d1c4 Nexthop local: fe80::6662:66ff:fe21:d1c4 BGP connection: shared network BGP Connect Retry Timer in Seconds: 120 Read thread: on Write thread: on FD used: 25 BGP neighbor is 10.250.100.2, remote AS 64513, local AS 64512, external link BGP version 4, remote router ID 10.250.100.2, local router ID 10.250.0.1 BGP state = Established, up for 01w5d06h Last read 00:00:11, Last write 00:00:09 Hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast Route refresh: advertised Address Family IPv4 Unicast: advertised and received Address Family IPv6 Unicast: received Hostname Capability: advertised (name: fj-rt.frijole.lol,domain name: n/a) not received Graceful Restart Capability: advertised Graceful restart information: Local GR Mode: Helper* Remote GR Mode: Disable R bit: False Timers: Configured Restart Time(sec): 120 Received Restart Time(sec): 0 Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 4 4 Notifications: 6 0 Updates: 23 4 Keepalives: 42661 42655 Route Refresh: 0 0 Capability: 0 0 Total: 42694 42663 Minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast Update group 5, subgroup 12 Packet Queue length 0 Community attribute sent to this neighbor(large) 1 accepted prefixes Connections established 4; dropped 3 Last reset 01w5d06h, No AFI/SAFI activated for peer Local host: 10.250.0.1, Local port: 179 Foreign host: 10.250.100.2, Foreign port: 51673 Nexthop: 10.250.0.1 Nexthop global: fdf6:5825:2efe:728b:6662:66ff:fe21:d1c4 Nexthop local: fe80::6662:66ff:fe21:d1c4 BGP connection: shared network BGP Connect Retry Timer in Seconds: 120 Read thread: on Write thread: on FD used: 28 BGP neighbor is 10.250.100.3, remote AS 64513, local AS 64512, external link BGP version 4, remote router ID 10.250.100.3, local router ID 10.250.0.1 BGP state = Established, up for 01w5d06h Last read 00:00:11, Last write 00:00:09 Hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast Route refresh: advertised Address Family IPv4 Unicast: advertised and received Address Family IPv6 Unicast: received Hostname Capability: advertised (name: fj-rt.frijole.lol,domain name: n/a) not received Graceful Restart Capability: advertised Graceful restart information: Local GR Mode: Helper* Remote GR Mode: Disable R bit: False Timers: Configured Restart Time(sec): 120 Received Restart Time(sec): 0 Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 4 4 Notifications: 6 0 Updates: 15 4 Keepalives: 42650 42643 Route Refresh: 0 0 Capability: 0 0 Total: 42675 42651 Minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast Update group 5, subgroup 12 Packet Queue length 0 Community attribute sent to this neighbor(large) 1 accepted prefixes Connections established 4; dropped 3 Last reset 01w5d06h, No AFI/SAFI activated for peer Local host: 10.250.0.1, Local port: 179 Foreign host: 10.250.100.3, Foreign port: 41147 Nexthop: 10.250.0.1 Nexthop global: fdf6:5825:2efe:728b:6662:66ff:fe21:d1c4 Nexthop local: fe80::6662:66ff:fe21:d1c4 BGP connection: shared network BGP Connect Retry Timer in Seconds: 120 Read thread: on Write thread: on FD used: 29 BGP neighbor is 10.250.100.4, remote AS 64513, local AS 64512, external link BGP version 4, remote router ID 10.250.100.4, local router ID 10.250.0.1 BGP state = Established, up for 01w5d06h Last read 00:00:23, Last write 00:00:21 Hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast Route refresh: advertised Address Family IPv4 Unicast: advertised and received Address Family IPv6 Unicast: received Hostname Capability: advertised (name: fj-rt.frijole.lol,domain name: n/a) not received Graceful Restart Capability: advertised Graceful restart information: Local GR Mode: Helper* Remote GR Mode: Disable R bit: False Timers: Configured Restart Time(sec): 120 Received Restart Time(sec): 0 Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 5 5 Notifications: 6 0 Updates: 23 5 Keepalives: 42667 42658 Route Refresh: 0 0 Capability: 0 0 Total: 42701 42668 Minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast Update group 5, subgroup 12 Packet Queue length 0 Community attribute sent to this neighbor(large) 1 accepted prefixes Connections established 5; dropped 4 Last reset 01w5d06h, No AFI/SAFI activated for peer Local host: 10.250.0.1, Local port: 179 Foreign host: 10.250.100.4, Foreign port: 48071 Nexthop: 10.250.0.1 Nexthop global: fdf6:5825:2efe:728b:6662:66ff:fe21:d1c4 Nexthop local: fe80::6662:66ff:fe21:d1c4 BGP connection: shared network BGP Connect Retry Timer in Seconds: 120 Read thread: on Write thread: on FD used: 26 BGP neighbor is 10.250.100.5, remote AS 64513, local AS 64512, external link BGP version 4, remote router ID 10.250.100.5, local router ID 10.250.0.1 BGP state = Established, up for 01w5d06h Last read 00:00:13, Last write 00:00:13 Hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: 4 Byte AS: advertised and received AddPath: IPv4 Unicast: RX advertised IPv4 Unicast Route refresh: advertised Address Family IPv4 Unicast: advertised and received Address Family IPv6 Unicast: received Hostname Capability: advertised (name: fj-rt.frijole.lol,domain name: n/a) not received Graceful Restart Capability: advertised Graceful restart information: Local GR Mode: Helper* Remote GR Mode: Disable R bit: False Timers: Configured Restart Time(sec): 120 Received Restart Time(sec): 0 Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 5 5 Notifications: 6 0 Updates: 24 5 Keepalives: 42662 42655 Route Refresh: 0 0 Capability: 0 0 Total: 42697 42665 Minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast Update group 5, subgroup 12 Packet Queue length 0 Community attribute sent to this neighbor(large) 1 accepted prefixes Connections established 5; dropped 4 Last reset 01w5d06h, No AFI/SAFI activated for peer Local host: 10.250.0.1, Local port: 179 Foreign host: 10.250.100.5, Foreign port: 46169 Nexthop: 10.250.0.1 Nexthop global: fdf6:5825:2efe:728b:6662:66ff:fe21:d1c4 Nexthop local: fe80::6662:66ff:fe21:d1c4 BGP connection: shared network BGP Connect Retry Timer in Seconds: 120 Read thread: on Write thread: on FD used: 27 -
@frijolierthnthou Were you able to solve this issue? I've been dealing with this same exact problem from the past 2 weeks