Routing Issue when Using 'WireGuard' interface group versus individual wireguard interfaces
-
So my network consists of two cloud PoPs sitting in Vultr, these are just lightweight Debian nodes running FRR and WireGuard, though I'm experimenting with replacing these with pfSense too at some point. On premise I have two pfSense nodes in a typical HA cluster. Each pfSense box has a WG point-to-point tunnel to each cloud PoP (WG0 to PoP1 and WG1 to Pop2). I have an IP Alias on localhost assigned a public IPv4 /32 address that I'm reserving from Vultr. Vultr allows you to announce reserved IPv4 addresses using BGP from multiple cloud instances in the same geographic region for redundancy. So, using FRR and OSPF, I'm announcing this address on-premise all the way up to Vultr and then redistributing OSPF into BGP to announce this address at the PoPs.
So basically what I have here is a highly available "IPv4" tunnel broker service. I'm doing this because I have an LTE backup modem on-premise that doesn't offer static addressing (Comcast Connection Pro) and using this technique I can failover and still have a static ingress point to a critical on-premise service...I use the term "critical" loosely here because this is all still very experimental.
Before pfSense was supporting Wireguard, I had two identical Debian VMs on-premise acting as WireGuard "shims" to sit between pfSense and the internet...pfSense supporting WireGuard has removed the need for these!
So far, this is working great. I can kill either cloud PoP, and OSPF / BGP handles the rerouting of the traffic through wireguard to the the active on-premise pfSense node...Though both cloud PoPs are hosted in the same Vultr location, I have them on different products (1 on Cloud Compute and 1 on High Frequency), just to add some extra assurance that both nodes aren't at least running on the same physical server.
This has actually been a very good test platform for teasing out a few bugs that I've been able to help squash in the process...MTU issues, gateway creation improvements, etc.
Anyways, I've got some else that's a bit 'weird' here.
So for the above scenario, I have assigned both WG0 and WG1 as OPT interfaces in pfSense. This creates the two gateways "WG0_GWV4" and "WG1_GWV4" that I then have in a gateway group to policy route the internal "critical" service. I'm using 1:1 NAT on both WG0 and WG1 to point the public /32 directly at the internal IPv4 address of the service. Again, this all works fine.
For testing purposes I have a simple pass all rule on both WG0 and WG1 OPT interfaces. However, when I created a test remote access wireguard tunnel and created a generic pass all rule on the built-in WireGuard Interface Group, this broke the scenario above. As soon as I delete the generic pass all rule on the WireGuard interface group, the scenario above starts working again. I can work around this by creating a more specific rule in the WireGuard interface group rules to only pass traffic sourced from the remote access tunnel subnet, but I still find it weird that a pass all rule in the wireguard group breaks things, but a pass all rule in the individual WG OPT interfaces doesn't.
-
@vbman213 said in Routing Issue when Using 'WireGuard' interface group versus individual wireguard interfaces:
For testing purposes I have a simple pass all rule on both WG0 and WG1 OPT interfaces. However, when I created a test remote access wireguard tunnel and created a generic pass all rule on the built-in WireGuard Interface Group, this broke the scenario above. As soon as I delete the generic pass all rule on the WireGuard interface group, the scenario above starts working again. I can work around this by creating a more specific rule in the WireGuard interface group rules to only pass traffic sourced from the remote access tunnel subnet, but I still find it weird that a pass all rule in the wireguard group breaks things, but a pass all rule in the individual WG OPT interfaces doesn't.
Rules on the group tab don't get
reply-toso return routing follows the routing table. Rules on the assigned interface tab getreply-toso packets matching those rules will exit back out the interface they entered.That's how it's always worked on any interface type, not new to WireGuard.