Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel

    WireGuard
    7
    30
    3.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      kevindd992002
      last edited by kevindd992002

      3c8cb3b7-8885-4843-9628-96a129719163-image.png

      I just switched from IPsec to WireGuard with pfsense 2.5 on both sides for my site-to-site tunnel. Traffic from the subnets on either side to the subnets on either side work just fine. Port forwarding (for inbound connections) and reply-to works beautifully as well. Gateway monitoring also works so the WG gateways on both sides are ONLINE.

      My problem is with PBR. It seems to be a hit and miss. As a test, on the local end I created a LAN rule for one client source IP to any destination and set it to use the WG gateway and on the remote end I set an outbound NAT rule for the whole source subnet (of the local end) on the remote end's WAN interface.

      I did a packet capture on the local end WG interface and saw the packets reaching that interface so that means that the packets are being routed correctly. However, when I do a packet capture on the remote end's WG interface with the same source IP, I don't see any results.

      This was all working properly with OpenVPN. Any ideas?

      P X 2 Replies Last reply Reply Quote 1
      • P
        pete35 @kevindd992002
        last edited by

        @kevindd992002

        just a guess: did you allow the interresting traffic in the correct firewall rules section for wireguard?

        <a href="https://carsonlam.ca">bintang88</a>
        <a href="https://carsonlam.ca">slot88</a>

        K 1 Reply Last reply Reply Quote 0
        • K
          kevindd992002 @pete35
          last edited by

          @pete35 said in Policy-based Routing through WG tunnel:

          @kevindd992002

          just a guess: did you allow the interresting traffic in the correct firewall rules section for wireguard?

          Oops, forgot to mention that yes I did. I have an any-any rule in the WG interface rules tab AND left the WG group tab rules empty. This is the recommended way of doing it so that reply-to's will work.

          1 Reply Last reply Reply Quote 0
          • K
            kevindd992002
            last edited by

            So I'm not sure if it's a PBR problem on 2.5 or a WG issue. I saw a recent thread in reddit regarding PBR issues for OpenVPN as well after upgrading to 2.5, if that matters.

            1 Reply Last reply Reply Quote 0
            • K
              kevindd992002
              last edited by

              Do you guys have any remote ideas for this? I'd really want to make this work just like OpenVPN.

              D H 2 Replies Last reply Reply Quote 1
              • D
                dma_pf @kevindd992002
                last edited by

                @kevindd992002 I haven't set up a site-to-site Wiregurad tunnel yet so I'm not going to be much help. But I thought I'd pass this along in case you haven't seen it. Hope it helps!
                https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html

                1 Reply Last reply Reply Quote 0
                • X
                  xxGBHxx @kevindd992002
                  last edited by

                  @kevindd992002 I have another thread on here about my problems with WG on PF. I have a PBR rule but I can't even get the tunnel stable enough to test the rule. If I do I'll come report how it goes.

                  G

                  1 Reply Last reply Reply Quote 1
                  • H
                    hypnosis4u2nv @kevindd992002
                    last edited by

                    @kevindd992002 Tom over at Lawrence Systems just put out a YouTube video for Wireguard site to site.

                    https://youtu.be/ZY49EAMnniY

                    1 Reply Last reply Reply Quote 0
                    • K
                      kevindd992002
                      last edited by

                      @dma_pf said in Policy-based Routing through WG tunnel:

                      @kevindd992002 I haven't set up a site-to-site Wiregurad tunnel yet so I'm not going to be much help. But I thought I'd pass this along in case you haven't seen it. Hope it helps!
                      https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-s2s.html

                      Thanks but I already read all the official documentation while I was setting this up. Coming from OpenVPN, then IPsec, and now WireGuard, I can say that WireGuard is very straightforward to setup so the chance of messing something up is lesser.

                      @xxgbhxx said in Policy-based Routing through WG tunnel:

                      @kevindd992002 I have another thread on here about my problems with WG on PF. I have a PBR rule but I can't even get the tunnel stable enough to test the rule. If I do I'll come report how it goes.

                      G

                      Thanks. I'll check your thread and maybe I can help. I have everything stable with WG except for PBR's.

                      @hypnosis4u2nv said in Policy-based Routing through WG tunnel:

                      @kevindd992002 Tom over at Lawrence Systems just put out a YouTube video for Wireguard site to site.

                      https://youtu.be/ZY49EAMnniY

                      I just watched the video. I have the exact same setup as his because one side is behind a CGNAT and the other side has a static public IP. I have the same settings except for a few things:

                      1. In the Peer WG Address field of each side's peer settings, I make sure to specify a single IP address without the CIDR (/32) notation. It works both ways because /32 is basically a single IP address also but it just makes it more simple if you use a single IP and this is what's documented in the official pfsense WG S2S article anyway.

                      2. I do not have keep alive on both sides enabled and it works just fine because of two things:
                        a. If the IP you set in the Peer WG Address field is included in the IP subnet settings of the actual WG interface, a gateway is automatically created and gateway monitoring is enabled by default. The monitoring itself is already the keep alive. This is documented here.
                        b. In the side with a static public IP, if you keep the Endpoint field blank this pfsense box WILL NOT initiate traffic to the remote peer until the remote peer sends traffic. This is documented here.

                      3. For gateway monitoring to work, the peer settings Allowed IPs field should contain the IP of the remote peer's WG interface on top of the subnets in the remote peer side that you want to route.

                      1 Reply Last reply Reply Quote 1
                      • K
                        kevindd992002
                        last edited by kevindd992002

                        I edited my OP and included a diagram of my setup. It's fairly a very basic setup and I have some additional weird observations with PBR and now port forwarding not working properly.

                        So again, for the topic related to PBR, here's what I have:

                        1. Site B LAN tab rules

                        1e5a4037-f649-40cb-b468-e8c2dca9d5b0-image.png

                        1. Site A Outbound NAT rules

                        ad93046d-f60b-4586-8c0d-abd795a1173a-image.png * Outbound NAT rule on WAN interface to translate packets with source IP = 192.168.20.0/24 to have a source IP = WAN address interface (static public IP)

                        Observations:

                        • 192.168.20.10 gets routed out Site A's WAN interface properly without any issues
                        • 192.168.20.11 does not get routed properly. I see packets reaching the Site B WG0 interface but I don't see anything in Site A's WG interface

                        Those PBR's are all for outbound. Now let's do inbound (port forwarding). Here's what I have on the topic related with port forwarding:

                        1. Site A Port Forward rules

                        dab50034-a7aa-48ab-9b99-d88968d70a93-image.png

                        Observations:

                        • I tested first using my usual external open port test site (tool1). Everything works as expected! I can reach both 192.168.20.10 and 192.168.20.11 (Site B clients) through Site A's WAN interface. So I thought everything was working properly.
                        • I then tested with another external open port test site (tool2) and now it's not working. The same exact behavior happens like the PBR issue above and that is the inbound packets reaches Site A's WG0 interface but stops there. Site B's WG0 interface never sees these packets.

                        So from my observations above, I can conclude that, for some odd reason, both PBR and port forwarding work with the "first" source IP and they don't with succeeding source IP's, if that even makes sense. The routing for both PBR and port forwarding is basically the same but only reverse of each other. This also proves that both pfsense boxes exhibit the problem.

                        @jimp @stephenw10 do you have any ideas?

                        1 Reply Last reply Reply Quote 0
                        • K
                          kevindd992002
                          last edited by kevindd992002

                          Workaround for the PBR issue

                          efd01098-4483-44e8-bdb4-4809592448b5-image.png

                          Workaround for the port forwarding issue

                          9aee3e7f-b1f3-4913-ab24-98bfc927a768-image.png I created WG0 interface outbound NAT rules to source translate inbound packets destined to 192.168.20.10:8989/192.168.20.11:8990 (from any) and it worked.

                          These should totally be unnecessary with WireGuard since it supports reply-to. It worked for the first local source IP (PBR issue) and external source IP test (port forwarding issue) which proves that reply-to does work. So there's got to be a bug somewhere here.

                          A 1 Reply Last reply Reply Quote 0
                          • A
                            AB5G @kevindd992002
                            last edited by

                            @kevindd992002 I have exactly the same issue here - https://forum.netgate.com/topic/160335/unsolved-possible-bug-wireguard-routing-weirdly/29
                            I'll try your fix and see if it works.

                            K 1 Reply Last reply Reply Quote 0
                            • K
                              kevindd992002 @AB5G
                              last edited by kevindd992002

                              @ab5g said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

                              @kevindd992002 I have exactly the same issue here - https://forum.netgate.com/topic/160335/unsolved-possible-bug-wireguard-routing-weirdly/29
                              I'll try your fix and see if it works.

                              Yeah, I did see your thread at one point when I was trying to reaearch on this issue. Let me know if my workaround solves your issue too. If it does, then I'd be inclined to say that this is a bug but I'll try capturing packets on the outer tunnel (WAN) now and see if I notice something there. @jimp one thing to point out here is that I can easily reproduce this behavior from both sides.

                              H A 2 Replies Last reply Reply Quote 0
                              • H
                                hypnosis4u2nv @kevindd992002
                                last edited by

                                @kevindd992002 https://youtu.be/mXG0RShQbaw

                                Not sure if this helps.

                                K 1 Reply Last reply Reply Quote 0
                                • K
                                  kevindd992002 @hypnosis4u2nv
                                  last edited by

                                  @hypnosis4u2nv said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

                                  @kevindd992002 https://youtu.be/mXG0RShQbaw

                                  Not sure if this helps.

                                  I just finished watching video but it's not applicable to the issue in this thread. The video explains routing through the tunnel from devices that have IP's in the transit network. This is the same as when I apply the outbound NAT workaround.

                                  Without the workaround, the source IP's do not get translated (both PBR andport forward issues) and that's where problem comes in. The routing works for only one source IP (the first ones I ever tested with).

                                  I hope that makes sense.

                                  H 1 Reply Last reply Reply Quote 1
                                  • A
                                    AB5G @kevindd992002
                                    last edited by

                                    @kevindd992002 That doesn't work for me. I'll put some more effort on it tomorrow.
                                    If you remove the /32 NAT rule and capture packets on the WG interface - do you see what I captured on my tcpdump (the source getting translated to the tunnel IP) or you don't even see that ?
                                    For me the NAT rule seemed to work, just that the packets don't leave the WAN for some reason.

                                    K 1 Reply Last reply Reply Quote 0
                                    • K
                                      kevindd992002 @AB5G
                                      last edited by kevindd992002

                                      @ab5g said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

                                      @kevindd992002 That doesn't work for me. I'll put some more effort on it tomorrow.
                                      If you remove the /32 NAT rule and capture packets on the WG interface - do you see what I captured on my tcpdump (the source getting translated to the tunnel IP) or you don't even see that ?
                                      For me the NAT rule seemed to work, just that the packets don't leave the WAN for some reason.

                                      Did you mean when keeping the /32 outbound NAT rule?

                                      1. With the NAT rule in place, yes I do see the source getting translated to the local WG interface IP and leaves the WAN and shows in the remote WG interface. Here's an example:

                                      Site B open state:

                                      85df323c-f187-4140-89e5-3b6b6f0bb4fe-image.png

                                      Site A open state:

                                      94983edc-1521-4b3f-a149-4c0b0f7b24d6-image.png

                                      That public IP is an external IP from a torrent peer.

                                      1. Without it, no SNAT happens and the packets do not leave the WAN interface.

                                      Site B open state:

                                      1b0db0cf-ad5e-409f-a334-a4b58dc54609-image.png

                                      Site A NO open state:

                                      82331efe-46d9-4cd0-a560-744e6cb3532a-image.png

                                      However, I noticed that from the same source and to different destination IP's through the tunnel, some are working and some are not! Here's proof of that:

                                      Site B open state:

                                      6e9101a3-7a13-44fa-9a3a-6034693fc67d-image.png

                                      Site A open state:

                                      efab353a-7e32-4b23-b997-a7ee1194756e-image.png

                                      So the problem seems to be "selective" or intermittent. It is not only isolated to one source IP working but other source IP's also work for different destination IP's, if that makes sense. And again, this weirdness is all solved with the outbound NAT rule workaround.

                                      A 1 Reply Last reply Reply Quote 0
                                      • H
                                        hypnosis4u2nv @kevindd992002
                                        last edited by

                                        @kevindd992002 no problem, and yes if makes sense. Hoping that watching tutorials brings some idea of a what may be causing the issue that your experiencing. I was able track down issues to what I was experiencing due to watching an somewhat unrelated topic turned out to be an option that I didn't enable.

                                        K 1 Reply Last reply Reply Quote 0
                                        • K
                                          kevindd992002 @hypnosis4u2nv
                                          last edited by

                                          @hypnosis4u2nv right, thanks. Which option fixed your issue and how did it fix it? I'm curious.

                                          H 1 Reply Last reply Reply Quote 0
                                          • A
                                            AB5G @kevindd992002
                                            last edited by

                                            @kevindd992002 said in Policy-based Routing (outbound) and port forwarding (inbound) through WG tunnel:

                                            So the problem seems to be "selective" or intermittent. It is not only isolated to one source IP working but other source IP's also work for different destination IP's, if that makes sense. And again, this weirdness is all solved with the outbound NAT rule workaround.

                                            That seems to be exactly my issue too - I don't know how to replicate it though. If you know how, can you raise a bug report please.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.