How to PBR over GRE tunnels?

thomas-worm

Hello,

I have a K8s cluster at Hetzner cloud including cloud virtual machines and dedicated bare metals connected via a vSwitch. I use a virtual machine with pfSense as a central entry point to the cluster, holding the public addresses on WAN. The k8s nodes are reachable via LAN interface, but are not next hop (Hetzner puts gateways for subnets in between). So I have to set up a GRE tunnel between pfSense and each node for routing issues. This are my addresses:

pfSense

WAN 1.1.1.1
LAN 10.0.0.10/16 with LAN gateway 10.10.0.1 (by Hetzner)
OPT1/GRE1 local 172.20.0.10 -> remote 172.20.0.100 (node 1)
OPT2/GRE2 local 172.20.0.10 -> remote 172.20.0.101 (node 2)

K8s node 1

LAN 10.10.0.100 with LAN gateway 10.0.0.1 (by Hetzner)
GRE1 local 172.20.0.100 -> remote 172.20.0.10 (pfSense)

K8s node 1

LAN 10.20.0.101 with LAN gateway 10.0.0.1 (by Hetzner)
GRE1 local 172.20.0.101 -> remote 172.20.0.10 (pfSense)

I want to setup MetalLB in the K8s cluster for mapping additional public IPs to K8s services.

My pfSense got the additional IP 2.2.2.2 on WAN.

Now comes the problem:
I was successful with adding an Virtual IP (ARP Proxy) for 2.2.2.2 and telling pfSense with a static route to use GRE1 (172.20.0.100) as gateway for routing 2.2.2.2. (Curl was successful getting "Hello world" from K8s service when getting http://2.2.2.2)
But the real intention is to use a gateway group and load balance the routing for 2.2.2.2 to the GRE 1 (172.20.0.100) and GRE2 (172.20.0.101) gateway. This is only possible with PBRs. Therefore I added a floating rule with direction "in" on wan, "quick" action and destination IP 2.2.2.2 routing to Gateway Group of GRE1 and GRE2. But when I try to call the Service (Curling http://2.2.2.2) the connection is neither refused not successfully conencting, it gets to a timeout. Even setting GRE1 as Gateway (without Gateway Group) results in a timeout. Why is the static route successful and PBR not? How can I route to a GRE as Gateway with PBRs?

Please don't tell to use HAproxy. I don't want to proxy the traffic, I want to route it for getting real MetalLB support in K8s.

Thanks for you help and best regards,
Thomas