Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel

kevindd992002 · Feb 23, 2021, 6:05 AM

I just switched from IPsec to WireGuard with pfsense 2.5 on both sides for my site-to-site tunnel. Traffic from the subnets on either side to the subnets on either side work just fine. Port forwarding (for inbound connections) and reply-to works beautifully as well. Gateway monitoring also works so the WG gateways on both sides are ONLINE.

My problem is with PBR. It seems to be a hit and miss. As a test, on the local end I created a LAN rule for one client source IP to any destination and set it to use the WG gateway and on the remote end I set an outbound NAT rule for the whole source subnet (of the local end) on the remote end's WAN interface.

I did a packet capture on the local end WG interface and saw the packets reaching that interface so that means that the packets are being routed correctly. However, when I do a packet capture on the remote end's WG interface with the same source IP, I don't see any results.

This was all working properly with OpenVPN. Any ideas?

pete35 · Feb 23, 2021, 6:38 AM

@kevindd992002

just a guess: did you allow the interresting traffic in the correct firewall rules section for wireguard?

kevindd992002 · Feb 23, 2021, 8:53 AM

@pete35 said in Policy-based Routing through WG tunnel:

@kevindd992002

just a guess: did you allow the interresting traffic in the correct firewall rules section for wireguard?

Oops, forgot to mention that yes I did. I have an any-any rule in the WG interface rules tab AND left the WG group tab rules empty. This is the recommended way of doing it so that reply-to's will work.

kevindd992002 · Feb 23, 2021, 8:54 AM

So I'm not sure if it's a PBR problem on 2.5 or a WG issue. I saw a recent thread in reddit regarding PBR issues for OpenVPN as well after upgrading to 2.5, if that matters.

kevindd992002 · Feb 25, 2021, 2:59 PM

Do you guys have any remote ideas for this? I'd really want to make this work just like OpenVPN.

dma_pf · Feb 25, 2021, 7:51 PM

@kevindd992002 I haven't set up a site-to-site Wiregurad tunnel yet so I'm not going to be much help. But I thought I'd pass this along in case you haven't seen it. Hope it helps!
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html

xxGBHxx · Feb 25, 2021, 11:19 PM

@kevindd992002 I have another thread on here about my problems with WG on PF. I have a PBR rule but I can't even get the tunnel stable enough to test the rule. If I do I'll come report how it goes.

G

hypnosis4u2nv · Feb 25, 2021, 11:46 PM

@kevindd992002 Tom over at Lawrence Systems just put out a YouTube video for Wireguard site to site.

https://youtu.be/ZY49EAMnniY

kevindd992002 · Feb 26, 2021, 3:58 AM

@dma_pf said in Policy-based Routing through WG tunnel:

@kevindd992002 I haven't set up a site-to-site Wiregurad tunnel yet so I'm not going to be much help. But I thought I'd pass this along in case you haven't seen it. Hope it helps!
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html

Thanks but I already read all the official documentation while I was setting this up. Coming from OpenVPN, then IPsec, and now WireGuard, I can say that WireGuard is very straightforward to setup so the chance of messing something up is lesser.

@xxgbhxx said in Policy-based Routing through WG tunnel:

@kevindd992002 I have another thread on here about my problems with WG on PF. I have a PBR rule but I can't even get the tunnel stable enough to test the rule. If I do I'll come report how it goes.

G

Thanks. I'll check your thread and maybe I can help. I have everything stable with WG except for PBR's.

@hypnosis4u2nv said in Policy-based Routing through WG tunnel:

@kevindd992002 Tom over at Lawrence Systems just put out a YouTube video for Wireguard site to site.

https://youtu.be/ZY49EAMnniY

I just watched the video. I have the exact same setup as his because one side is behind a CGNAT and the other side has a static public IP. I have the same settings except for a few things:

In the Peer WG Address field of each side's peer settings, I make sure to specify a single IP address without the CIDR (/32) notation. It works both ways because /32 is basically a single IP address also but it just makes it more simple if you use a single IP and this is what's documented in the official pfsense WG S2S article anyway.
I do not have keep alive on both sides enabled and it works just fine because of two things:
a. If the IP you set in the Peer WG Address field is included in the IP subnet settings of the actual WG interface, a gateway is automatically created and gateway monitoring is enabled by default. The monitoring itself is already the keep alive. This is documented here.
b. In the side with a static public IP, if you keep the Endpoint field blank this pfsense box WILL NOT initiate traffic to the remote peer until the remote peer sends traffic. This is documented here.
For gateway monitoring to work, the peer settings Allowed IPs field should contain the IP of the remote peer's WG interface on top of the subnets in the remote peer side that you want to route.

kevindd992002 · Feb 26, 2021, 6:57 AM

I edited my OP and included a diagram of my setup. It's fairly a very basic setup and I have some additional weird observations with PBR and now port forwarding not working properly.

So again, for the topic related to PBR, here's what I have:

Site B LAN tab rules

login-to-view

Site A Outbound NAT rules

login-to-view * Outbound NAT rule on WAN interface to translate packets with source IP = 192.168.20.0/24 to have a source IP = WAN address interface (static public IP)

Observations:

192.168.20.10 gets routed out Site A's WAN interface properly without any issues
192.168.20.11 does not get routed properly. I see packets reaching the Site B WG0 interface but I don't see anything in Site A's WG interface

Those PBR's are all for outbound. Now let's do inbound (port forwarding). Here's what I have on the topic related with port forwarding:

Site A Port Forward rules

login-to-view

Observations:

I tested first using my usual external open port test site (tool1). Everything works as expected! I can reach both 192.168.20.10 and 192.168.20.11 (Site B clients) through Site A's WAN interface. So I thought everything was working properly.
I then tested with another external open port test site (tool2) and now it's not working. The same exact behavior happens like the PBR issue above and that is the inbound packets reaches Site A's WG0 interface but stops there. Site B's WG0 interface never sees these packets.

So from my observations above, I can conclude that, for some odd reason, both PBR and port forwarding work with the "first" source IP and they don't with succeeding source IP's, if that even makes sense. The routing for both PBR and port forwarding is basically the same but only reverse of each other. This also proves that both pfsense boxes exhibit the problem.

@jimp @stephenw10 do you have any ideas?

kevindd992002 · Feb 26, 2021, 8:09 AM

Workaround for the PBR issue

login-to-view

Workaround for the port forwarding issue

login-to-view I created WG0 interface outbound NAT rules to source translate inbound packets destined to 192.168.20.10:8989/192.168.20.11:8990 (from any) and it worked.

These should totally be unnecessary with WireGuard since it supports reply-to. It worked for the first local source IP (PBR issue) and external source IP test (port forwarding issue) which proves that reply-to does work. So there's got to be a bug somewhere here.

AB5G · Feb 28, 2021, 2:04 AM

@kevindd992002 I have exactly the same issue here - https://forum.netgate.com/topic/160335/unsolved-possible-bug-wireguard-routing-weirdly/29
I'll try your fix and see if it works.

kevindd992002 · Feb 28, 2021, 2:07 AM

@ab5g said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

@kevindd992002 I have exactly the same issue here - https://forum.netgate.com/topic/160335/unsolved-possible-bug-wireguard-routing-weirdly/29
I'll try your fix and see if it works.

Yeah, I did see your thread at one point when I was trying to reaearch on this issue. Let me know if my workaround solves your issue too. If it does, then I'd be inclined to say that this is a bug but I'll try capturing packets on the outer tunnel (WAN) now and see if I notice something there. @jimp one thing to point out here is that I can easily reproduce this behavior from both sides.

hypnosis4u2nv · Feb 28, 2021, 2:18 AM

@kevindd992002 https://youtu.be/mXG0RShQbaw

Not sure if this helps.

kevindd992002 · Feb 28, 2021, 3:05 AM

@hypnosis4u2nv said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

@kevindd992002 https://youtu.be/mXG0RShQbaw

Not sure if this helps.

I just finished watching video but it's not applicable to the issue in this thread. The video explains routing through the tunnel from devices that have IP's in the transit network. This is the same as when I apply the outbound NAT workaround.

Without the workaround, the source IP's do not get translated (both PBR andport forward issues) and that's where problem comes in. The routing works for only one source IP (the first ones I ever tested with).

I hope that makes sense.

AB5G · Feb 28, 2021, 3:08 AM

@kevindd992002 That doesn't work for me. I'll put some more effort on it tomorrow.
If you remove the /32 NAT rule and capture packets on the WG interface - do you see what I captured on my tcpdump (the source getting translated to the tunnel IP) or you don't even see that ?
For me the NAT rule seemed to work, just that the packets don't leave the WAN for some reason.

kevindd992002 · Feb 28, 2021, 5:40 AM

@ab5g said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

@kevindd992002 That doesn't work for me. I'll put some more effort on it tomorrow.
If you remove the /32 NAT rule and capture packets on the WG interface - do you see what I captured on my tcpdump (the source getting translated to the tunnel IP) or you don't even see that ?
For me the NAT rule seemed to work, just that the packets don't leave the WAN for some reason.

Did you mean when keeping the /32 outbound NAT rule?

With the NAT rule in place, yes I do see the source getting translated to the local WG interface IP and leaves the WAN and shows in the remote WG interface. Here's an example:

Site B open state:

login-to-view

Site A open state:

login-to-view

That public IP is an external IP from a torrent peer.

Without it, no SNAT happens and the packets do not leave the WAN interface.

Site B open state:

login-to-view

Site A NO open state:

login-to-view

However, I noticed that from the same source and to different destination IP's through the tunnel, some are working and some are not! Here's proof of that:

Site B open state:

login-to-view

Site A open state:

login-to-view

So the problem seems to be "selective" or intermittent. It is not only isolated to one source IP working but other source IP's also work for different destination IP's, if that makes sense. And again, this weirdness is all solved with the outbound NAT rule workaround.

hypnosis4u2nv · Feb 28, 2021, 6:55 AM

@kevindd992002 no problem, and yes if makes sense. Hoping that watching tutorials brings some idea of a what may be causing the issue that your experiencing. I was able track down issues to what I was experiencing due to watching an somewhat unrelated topic turned out to be an option that I didn't enable.

kevindd992002 · Feb 28, 2021, 7:43 AM

@hypnosis4u2nv right, thanks. Which option fixed your issue and how did it fix it? I'm curious.

AB5G · Feb 28, 2021, 12:53 PM

@kevindd992002 said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

So the problem seems to be "selective" or intermittent. It is not only isolated to one source IP working but other source IP's also work for different destination IP's, if that makes sense. And again, this weirdness is all solved with the outbound NAT rule workaround.

That seems to be exactly my issue too - I don't know how to replicate it though. If you know how, can you raise a bug report please.