Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PFsense as OpenVPN Client - Networks can't be reached

    OpenVPN
    2
    6
    494
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • O
      Orwi
      last edited by Orwi

      Hi,
      I tried all I can, but don't get any further, so here I am.
      Any help regards a solution or better analysis is welcome, I am EndOfWisdom.

      The Scenario
      A Site-To-Site OpenVPN connection, where my main firewall is the VPN Client.

      The Problem
      I can't reach any other client IP other than the PFSense OpenVPN ClientIP from server-side. Otherwise I am able to contact the other networks of the server from the clients without any headache.

      General settings
      You can safely assume, that every firewall tab has some allow all - as I am debugging this (unsuccessfully) quite a time.

      Configs -VPN
      Server - Ingress (OPNsense)
      Network Settings
      HADES (ovpns1) -> v4: 192.168.250.1/24
      WAN (vtnet0) -> v4: publicIP/24
      testLAN (lo1) -> v4: 10.1.1.1/24

      OpenVPN Settings
      The VPN connection in general works 'flawless', except of the clients subnets routing problems

      Server - ingress (OPNSense)
      General Information
      Server Mode: Remote Access (SSL/TLS)
      Protocol: UDP4
      Device Mode: tun
      Interface: WAN

      Tunnel Settings
      IPv4 Tunnel Network: 192.168.250.0/24
      IPv4 Local Network: 10.1.1.0/24
      IPv4 Remote Network: 10.10.10.10/32,10.10.100.0/22
      Concurrent connections: 1
      Disable IPv6: check

      Client Settings
      Dynamic IP: check
      Address Pool: check
      Topology: check

      Advanced configuration
      tun-mtu 1500
      mssfix 1500

      Client - Hades (PFsense)
      OpenVPN Settings
      The VPN connection in general works 'flawless', except of the clients subnets routing problems

      General Information
      Server Mode: Remote Access (SSL/TLS)
      Protocol: UDP4
      Device Mode: tun
      Interface: WAN

      Tunnel Settings
      IPv4 Tunnel Network: 192.168.250.0/24
      IPv4 Remote Network: 10.1.1.0/24
      IPv4 Local Network !! NOT available in options !! Would be: 10.10.10.10/32,10.10.100.0/22
      Topology: Subnet -- One IP address per client in a common subnet

      Advanced configuration
      tun-mtu 1500
      mssfix 1500

      Routes Server
      root@ingress:~ # netstat -r
      Routing tables

      Internet:
      Destination Gateway Flags Netif Expire
      default pupIP UGS vtnet0
      ingress link#6 UH lo1
      10.10.10.5/32 192.168.250.2 UGS ovpns1
      10.10.10.7/32 192.168.250.2 UGS ovpns1
      10.10.10.10/32 192.168.250.2 UGS ovpns1
      10.10.100.0/22 192.168.250.2 UGS ovpns1
      localhost link#3 UH lo0
      192.168.250.0/24 192.168.250.2 UGS ovpns1
      ingress link#7 UHS lo0
      192.168.250.2 link#7 UH ovpns1
      publicIP/24 link#1 U vtnet0
      ingress link#1 UHS lo0

      Routes Client

      Destination Gateway Flags Use Mtu Netif Expire
      default pupIP UGS 950434 1500 igb0
      10.1.1.0/24 192.168.250.1 UGS 0 1500 ovpnc5
      10.10.10.1 link#2 UHS 0 16384 lo0
      10.10.10.1/32 link#2 U 0 1500 igb1
      10.10.10.5 link#10 UHS 0 16384 lo0
      10.10.10.5/32 link#10 U 0 1500 igb2.100
      10.10.10.7 link#10 UHS 342897 16384 lo0
      10.10.10.7/32 link#10 U 0 1500 igb2.100
      10.10.10.8 link#11 UHS 0 16384 lo0
      10.10.10.8/32 link#11 U 0 1500 igb2.200
      10.10.10.9 link#11 UHS 0 16384 lo0
      10.10.10.9/32 link#11 U 0 1500 igb2.200
      10.10.10.10 link#10 UHS 0 16384 lo0
      10.10.10.10/32 link#10 U 0 1500 igb2.100
      10.10.10.11 link#9 UHS 0 16384 lo0
      10.10.10.11/32 link#9 U 0 1500 igb2.10
      10.10.100.0/22 link#12 U 4278819 1500 igb2.1000
      192.168.250.1 link#17 UH 7795 1500 ovpnc5
      192.168.250.2 link#17 UHS 0 16384 lo0

      TCPdump serverside WAN
      The server sents to the correct route, but.. retransmission, doesn't get anything. Tested with port forward on 444

      1 0.000000 anotherPublicDialUpIP ingressPermIP TCP 74 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
      2 0.151688 anotherPublicDialUpIP ingressPermIP TCP 74 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
      3 0.921882 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
      4 1.160898 anotherPublicDialUpIP ingressPermIP TCP 74 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

      TCPdump serverside VPN
      1 0.000000 anotherPublicDialUpIP 10.10.10.10 TCP 64 58777 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417955 TSecr=0 WS=512
      2 0.151733 anotherPublicDialUpIP 10.10.10.10 TCP 64 37745 → 444 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10417980 TSecr=0 WS=512
      3 0.921826 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 58777 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418056 TSecr=0 WS=512
      4 1.160824 anotherPublicDialUpIP 10.10.10.10 TCP 64 [TCP Retransmission] 37745 → 444 [SYN] Seq=0 Win=65535 Len=0 MSS=1360 SACK_PERM=1 TSval=10418080 TSecr=0 WS=512

      TCPdump hades clientside
      nothing

      Rough network diagram
      550e5683-b49b-4f01-ad29-fcf538850402-image.png

      possible solutions and workaround
      The goal of this is, to forward incoming requests to the ingressPermIP to servers in the segments of hades (vpn client)
      For trivial web, I could go with 2 HAProxies so, the webrequest goes from ingress->hadesVpnIP->server via proxy protocol. Done and tested.

      But sadly this is not for every and all services a solution. And it doesn't solve the problem in general.

      some random observations
      traceroute is nearly useless for me. Don't know why, but even working path end up just with stars until given up after 30 hops.
      Currently I am trying to rebuild the TTL error, I was able to produce instead just a timeout with ping.

      ecosystem
      the pfsense is baremetal on an pcengine
      the opnsense is a virtualized "cloud node", I have no further access or information to.

      Did I forgot something? Just ask and I am eager to give the information. This is really burning my time. :(

      V 1 Reply Last reply Reply Quote 0
      • O
        Orwi
        last edited by Orwi

        It may be worth to mention, that I am routing some 'LAN' device through the VPN to use the permIP as gateway - mainly for testing. (policy-routing via src of device)
        THIS works great, ever since.
        And also setups with the pfsense as server have no known problems. To be specific the "testdevice" is dialed in via vpn in hades and then routed through the vpn where hades is the client and exits ingress.

        1 Reply Last reply Reply Quote 0
        • O
          Orwi
          last edited by

          I made a simpler setup pfsense instance with to the minmal reduced base config and the same behavior.
          It has nearly no additional packages, except HAProxy and ACME, only 1 virtual IP on LAN side and the same behavior.

          Next step: a clean instance of PFsense. Will report back. And also test this with an opnsense instance on local side.

          1 Reply Last reply Reply Quote 0
          • O
            Orwi
            last edited by

            Setup different OPNsense and PFsense boxes for better insights.
            Turns out, no problem if only two pfsense are involved.

            The above described problems also exists with vanilla OPNsense boxes.

            Beside switching the server site from opn to pfsense, nothing else was changed, but the subnets are now reachable!

            1 Reply Last reply Reply Quote 0
            • O
              Orwi
              last edited by

              @PFSense Team
              If I build a site to site (peer to peer) with SSL/TLS it is shown as client instance in openVPN status. Also the routing does not work, alike described above.
              Expected behavior is, that it is also shown in Peer to Peer Server Instance Statistics not in Client Instance Statistics

              If I use a shared key site to site (peer to peer), than everything works as expected*.

              • except my forwarded packages enters as expected and reach the destination BUT leaves via WAN instead of VPN**.

              ** which is also a gateway for policy based routing for other clients. Could this be a/the problem?

              Also the documentation is flawed: https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-s2s-tls.html

              It may be a minor mistake, still IPv4 Remote Network is addressed twice. This really decreases the trust in such documentation.

              1 Reply Last reply Reply Quote 0
              • V
                viragomann @Orwi
                last edited by viragomann

                @Orwi said in PFsense as OpenVPN Client - Networks can't be reached:

                A Site-To-Site OpenVPN connection
                IPv4 Tunnel Network: 192.168.250.0/24
                Concurrent connections: 1

                If it is a site to site vpn and only 1 connection is allowed, why using a /24 tunnel. Set it to /30.

                Advanced configuration
                tun-mtu 1500
                mssfix 1500

                Be careful with these settings.

                @Orwi said in PFsense as OpenVPN Client - Networks can't be reached:

                except my forwarded packages enters as expected and reach the destination BUT leaves via WAN instead of VPN**.

                ** which is also a gateway for policy based routing for other clients. Could this be a/the problem?

                No.

                So you have already assigned interfaces to the OpenVPN instances?

                Ensure to add a firewall rule allowing the desired access to that interface on the incoming site and that this rule is applied.
                There must not be a rule on the OpenVPN or on floating tab which matches to that traffic!

                If you're unsure which rule is applied enable logging and check the logs after testing.

                @Orwi said in PFsense as OpenVPN Client - Networks can't be reached:

                Also the documentation is flawed: https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-s2s-tls.html
                It may be a minor mistake, still IPv4 Remote Network is addressed twice.

                ??

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post