Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense Multi WAN Site-to-Site OpenVPN Tunnel Port Forward Routing Issue

    Scheduled Pinned Locked Moved
    Routing and Multi WAN
    2
    4
    685
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • SgtKilgore406S
      SgtKilgore406
      last edited by

      First off, I apologize if this is the wrong sub category for this question but I believe the issue is with routing and multiple WANs are involved. I have shared a rough network diagram of what I am trying to accomplish at the bottom of this post. I think this is a very simple problem with a simple solution but I am racking my brain trying to solve it.

      A. Preface:

      1. This is a private, not corporate network.
      2. I have dual WAN setup with Starlink as my primary ISP and another local ADSL ISP. This is primarily for automatic failover since Starlink is still "Better Than Nothing Beta".
      3. Starlink is using CGNAT so port forwarding is out of the question until they get IPv6 into production.
      4. The local ADSL ISP offers static public IPs to which I have one. The problem here is the upload speed is at a maximum of 1Mbps. Most of the time it is less.
      5. Both local and cloud hosted pfSense systems are running 2.5.0. After having issues with port forwarding on 2.5.1 on my local FW (apparently a known bug) I reverted the software which fixed that issue.

      B. Goal:

      What I am trying to accomplish is to use a cloud hosted pfSense box to accept public connection attempts and route the port forwards across an OpenVPN Site to Site Server/Client tunnel back to my primary firewall where the servers reside. The intention is that both inbound and outbound traffic would flow across this tunnel.

      The reason for this endeavor is to increase the upload speed when using personal services, like my Plex server, outside the local network.

      C. Current Status

      The cloud hosted pfSense FW is online and I have a successful tunnel connection between both firewalls. I am able to ping from both sides through the tunnel. I have already updated my public DNS records to point to the cloud hosted IP instead of my local ISP.

      D. Problem I Am Trying To Solve

      After pulling my hair out for a while I have found that my local firewall is receiving network traffic as expected from the cloud hosted pfSense box. It appears that the reply traffic from my servers is being routed out my Starlink gateway instead of traveling back through the tunnel to the cloud pfSense box.

      The output below is from a packet capture, on my local firewall, from the local vpn tunnel interface.

      10:32:37.358177 IP 107.77.201.77.4718 > 10.1.1.8.443: tcp 0
10:32:37.498171 IP 107.77.201.77.49843 > 10.1.1.8.443: tcp 0
10:32:37.498195 IP 107.77.201.77.51667 > 10.1.1.8.443: tcp 0
10:32:37.498279 IP 107.77.201.77.53320 > 10.1.1.8.443: tcp 0
10:32:40.362141 IP 107.77.201.77.4718 > 10.1.1.8.443: tcp 0
10:32:40.502114 IP 107.77.201.77.51667 > 10.1.1.8.443: tcp 0
10:32:40.502234 IP 107.77.201.77.53320 > 10.1.1.8.443: tcp 0
10:32:40.502240 IP 107.77.201.77.49843 > 10.1.1.8.443: tcp 0
10:32:41.966967 IP 107.77.201.77.29209 > 10.1.1.8.443: tcp 0
10:32:43.567228 IP 107.77.201.77.4718 > 10.1.1.8.443: tcp 0
10:32:43.710891 IP 107.77.201.77.51667 > 10.1.1.8.443: tcp 0
10:32:43.710904 IP 107.77.201.77.53320 > 10.1.1.8.443: tcp 0
10:32:43.710910 IP 107.77.201.77.49843 > 10.1.1.8.443: tcp 0
10:32:44.966918 IP 107.77.201.77.29209 > 10.1.1.8.443: tcp 0
10:32:46.782792 IP 107.77.201.77.4718 > 10.1.1.8.443: tcp 0
10:32:46.918796 IP 107.77.201.77.51667 > 10.1.1.8.443: tcp 0
10:32:46.918806 IP 107.77.201.77.53320 > 10.1.1.8.443: tcp 0
10:32:46.918897 IP 107.77.201.77.49843 > 10.1.1.8.443: tcp 0
10:32:47.806808 IP 107.77.201.77.26663 > 10.1.1.8.443: tcp 0
10:32:47.806854 IP 107.77.201.77.29779 > 10.1.1.8.443: tcp 0
10:32:47.806874 IP 107.77.201.77.27233 > 10.1.1.8.443: tcp 0
10:32:48.182761 IP 107.77.201.77.29209 > 10.1.1.8.443: tcp 0
10:32:48.286855 IP 107.77.201.77.20659 > 10.1.1.8.443: tcp 0

      This next output below is from my Starlink gateway (default gateway on the local firewall).

      10:35:45.971012 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:46.131077 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:46.131079 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:46.979068 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:47.139008 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:47.139008 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:48.970992 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:49.131088 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:49.131089 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:49.331073 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:49.331075 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:50.339017 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:50.339019 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:50.979004 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:51.138974 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:51.138974 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:52.179020 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:52.330941 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:52.330943 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:52.338863 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:52.338867 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:52.531233 IP 10.1.1.8.443 > 107.77.201.77.34890: tcp 0
10:35:53.538928 IP 10.1.1.8.443 > 107.77.201.77.34890: tcp 0
10:35:54.338989 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:54.338991 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:55.394929 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0
10:35:55.530942 IP 10.1.1.8.443 > 107.77.201.77.34890: tcp 0
10:35:55.538806 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:55.538807 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:55.554717 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:55.554826 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:56.451057 IP 10.1.1.8.443 > 107.77.201.77.24706: tcp 0
10:35:56.451059 IP 10.1.1.8.443 > 107.77.201.77.62288: tcp 0
10:35:57.474983 IP 10.1.1.8.443 > 107.77.201.77.24706: tcp 0
10:35:57.474985 IP 10.1.1.8.443 > 107.77.201.77.62288: tcp 0
10:35:57.539010 IP 10.1.1.8.443 > 107.77.201.77.34890: tcp 0
10:35:58.746891 IP 10.1.1.8.443 > 107.77.201.77.34890: tcp 0
10:35:58.754791 IP 10.1.1.8.443 > 107.77.201.77.38956: tcp 0
10:35:58.754793 IP 10.1.1.8.443 > 107.77.201.77.29342: tcp 0
10:35:59.450824 IP 10.1.1.8.443 > 107.77.201.77.24706: tcp 0
10:35:59.450825 IP 10.1.1.8.443 > 107.77.201.77.62288: tcp 0
10:35:59.618932 IP 10.1.1.8.443 > 107.77.201.77.49577: tcp 0
10:35:59.618935 IP 10.1.1.8.443 > 107.77.201.77.49995: tcp 0
10:35:59.618937 IP 10.1.1.8.443 > 107.77.201.77.48608: tcp 0

      E. What I Have Tried

      1. I have tried configuring the DMZ to use the VPN tunnel as its default gateway along with trying an individual host.
        adf9ba70-8f4e-4597-ad2c-520c9aaf9cc6-image.png
        e7869f45-d6c4-47c5-aa04-b4c9d34ef929-image.png
        The gateway firewall rule is working to some extent because any traffic generated from a system on the DMZ is directed out through the VPN tunnel so I am confused as to why the external connection replies are not traversing back through the tunnel.

      2. Selecting "Do not NAT" on the DMZ rules for Starlink's outbound NAT and disabling the DMZ rules made no difference. The packet captures still show the traffic exiting the Starlink gateway.

      F. Summary

      I believe this is an easy fix but I am not having any luck finding it. Any help would be greatly appreciated!

      G. Network Diagram

      Starlink_Port_Forwarding_Dilemma_5-22-2021.png

      V 1 Reply Last reply Reply Quote 0
      • V
        viragomann @SgtKilgore406
        last edited by

        @sgtkilgore406 said in pfSense Multi WAN Site-to-Site OpenVPN Tunnel Port Forward Routing Issue:

        I think this is a very simple problem with a simple solution but I am racking my brain trying to solve it.

        Pretty much text for a simple problem, I think.

        E. What I Have Tried
        I have tried configuring the DMZ to use the VPN tunnel as its default gateway along with trying an individual host.

        This only affects outbound traffic, not inbound.

        Selecting "Do not NAT" on the DMZ rules for Starlink's outbound NAT and disabling the DMZ rules made no difference. The packet captures still show the traffic exiting the Starlink gateway.

        This also affects only outbound and simply disables NAT but does not prevent the packets to be routed to the default gateway.

        What you should do:

        If you didn't already, assign an interface to the OpenVPN site-to-site instance on the local pfSense.

        Then move firewall rules allowing incoming traffic to this interface.
        Remove all pass rules from the OpenVPN tab!

        This doesn't work on 2.5.1, but it should on 2.5.0.

        SgtKilgore406S 1 Reply Last reply Reply Quote 1
        • SgtKilgore406S
          SgtKilgore406 @viragomann
          last edited by

          @viragomann said in pfSense Multi WAN Site-to-Site OpenVPN Tunnel Port Forward Routing Issue:

          What you should do:

          If you didn't already, assign an interface to the OpenVPN site-to-site instance on the local pfSense.

          Then move firewall rules allowing incoming traffic to this interface.
          Remove all pass rules from the OpenVPN tab!

          This doesn't work on 2.5.1, but it should on 2.5.0.

          Removing all the rules from the OpenVPN tab fixed the routing issue! I already had a virtual interface created for the site-to-site connection.

          The OpenVPN tab in the Firewall rules was being used for my remote access (RA) OpenVPN configuration. I have created a virtual interface for it and created rules but the RA VPN appears to be broken. I'll fool around with the RA VPN later and try to get it fixed. If need be I'll change the RA VPN over to IPsec .

          Thank you very much @viragomann! This made my day.

          V 1 Reply Last reply Reply Quote 0
          • V
            viragomann @SgtKilgore406
            last edited by

            @sgtkilgore406 said in pfSense Multi WAN Site-to-Site OpenVPN Tunnel Port Forward Routing Issue:

            I have created a virtual interface for it and created rules but the RA VPN appears to be broken. I'll fool around with the RA VPN later and try to get it fixed.

            It should work this way though. It doesn't matter if the rules resides on the interface tab or on OpenVPN. The OpenVPN is just an interface group including all OpenVPN instances running on the box and is added when the first one is set up.

            1 Reply Last reply Reply Quote 0
            • First post
              Last post