Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Route WAN network to OVPN

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 2 Posters 1.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      DasK
      last edited by DasK

      Hello,

      I have a proxmox cloud server with a VM pfSense.

      Proxmox server: 10.1.1.1/30
      PfSense WAN: 10.1.1.2/30
      PfSense LAN: 192.168.10.0/24

      The pfSense is connected to a remote VPN server (on another pfSense server)
      VPN: 10.23.1.1/24

      I added an outbound rule to redirect WAN to VPN interface (if I'm not wrong)
      f33e1a83-0de3-4e2e-b507-350d24d568bf-image.png

      Now from the pfSense itself, I'm able to ping/traceroute/rpcinfo a server I have inside the VPN tunnel from the WAN interface, IP: 10.23.1.100
      5f70ecf6-93d7-4431-bfcb-b22a273f3e4d-image.png
      62fe640f-abc2-4db2-89d8-83efe8273af9-image.png

      Now, I switch to proxmox server, I added the route to the VPN

      Now the strange thing start, I'm able to ping the remote server (10.23.1.100), but not able to fully traceroute it or rpcinfo

      d9d062de-e0bd-4a94-ab4b-bf2370703c84-image.png

      The iptables rules are correct (tried with full allow rules)
      Inside pfSense, I allow traffic from * to * on all interfaces.

      It seems like my server can communicate to it, but I don't have the answer but the strange thing is that the pfSense WAN 10.1.1.2 can do it, but not my proxmox server 10.1.1.1 ..

      Thanks for your help

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        I assume OPT1 is the assigned OpenVPN interface?

        Check the state table when you are trying to connect or successfully pinging. Is the traffic NAT'd to the OPT1 interface IP as expected?

        Run a packet capture on the OpenVPN interface. What actually being sent from there?

        Steve

        1 Reply Last reply Reply Quote 0
        • D
          DasK
          last edited by DasK

          I done the Packet Capture on the OPT1 interface while doing a ping and a traceroute on the proxmox serveur

          15:09:18.758929 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 24999, length 8
15:09:18.768983 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 24999, length 8
15:09:19.289009 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25000, length 8
15:09:19.299123 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25000, length 8
15:09:19.830176 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25001, length 8
15:09:19.840291 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25001, length 8
15:09:20.371708 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25002, length 8
15:09:20.381646 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25002, length 8
15:09:20.888891 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25003, length 8
15:09:20.898926 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25003, length 8
15:09:21.418876 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 1, length 64
15:09:21.419019 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25004, length 8
15:09:21.428766 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25004, length 8
15:09:21.441468 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 1, length 64
15:09:21.937154 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25005, length 8
15:09:21.946987 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25005, length 8
15:09:22.420103 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 2, length 64
15:09:22.442994 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 2, length 64
15:09:22.448187 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25006, length 8
15:09:22.458130 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25006, length 8
15:09:22.955888 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25007, length 8
15:09:22.965827 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25007, length 8
15:09:23.420449 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 3, length 64
15:09:23.443246 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 3, length 64
15:09:23.489014 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25008, length 8
15:09:23.498938 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25008, length 8
15:09:24.030321 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25009, length 8
15:09:24.040293 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25009, length 8
15:09:24.545142 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25010, length 8
15:09:24.555173 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25010, length 8
15:09:25.058926 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25011, length 8
15:09:25.068957 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25011, length 8
15:09:25.537847 IP 10.23.1.200.35330 > 10.23.1.100.33437: UDP, length 32
15:09:25.537863 IP 10.23.1.200.18025 > 10.23.1.100.33438: UDP, length 32
15:09:25.537871 IP 10.23.1.200.2270 > 10.23.1.100.33439: UDP, length 32
15:09:25.537879 IP 10.23.1.200.23400 > 10.23.1.100.33440: UDP, length 32
15:09:25.537886 IP 10.23.1.200.39105 > 10.23.1.100.33441: UDP, length 32
15:09:25.537894 IP 10.23.1.200.14561 > 10.23.1.100.33442: UDP, length 32
15:09:25.537917 IP 10.23.1.200.5445 > 10.23.1.100.33443: UDP, length 32
15:09:25.537925 IP 10.23.1.200.14700 > 10.23.1.100.33444: UDP, length 32
15:09:25.537933 IP 10.23.1.200.1751 > 10.23.1.100.33445: UDP, length 32
15:09:25.537954 IP 10.23.1.200.49371 > 10.23.1.100.33446: UDP, length 32
15:09:25.537961 IP 10.23.1.200.35188 > 10.23.1.100.33447: UDP, length 32
15:09:25.537969 IP 10.23.1.200.18223 > 10.23.1.100.33448: UDP, length 32
15:09:25.537992 IP 10.23.1.200.56560 > 10.23.1.100.33449: UDP, length 32
15:09:25.540117 IP 10.23.1.200.14513 > 10.23.1.100.33450: UDP, length 32
15:09:25.540135 IP 10.23.1.200.34198 > 10.23.1.100.33451: UDP, length 32
15:09:25.540147 IP 10.23.1.200.40572 > 10.23.1.100.33452: UDP, length 32
15:09:25.548161 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:09:25.548208 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:09:25.548233 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:09:25.550430 IP 10.23.1.200.44090 > 10.23.1.100.33453: UDP, length 32
15:09:25.550444 IP 10.23.1.200.30073 > 10.23.1.100.33454: UDP, length 32
15:09:25.550453 IP 10.23.1.200.46325 > 10.23.1.100.33455: UDP, length 32
15:09:25.588888 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25012, length 8
15:09:25.598760 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25012, length 8
15:09:26.130202 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25013, length 8
15:09:26.140134 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25013, length 8
15:09:26.658825 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25014, length 8
15:09:26.668659 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25014, length 8
15:09:27.180239 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25015, length 8
15:09:27.190104 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25015, length 8
15:09:27.705401 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25016, length 8
15:09:27.715239 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25016, length 8
15:09:28.220925 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25017, length 8
15:09:28.231023 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25017, length 8
15:09:28.754789 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25018, length 8
15:09:28.764623 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25018, length 8
15:09:29.273490 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25019, length 8

          The ip: 10.23.1.200 is the IP of the proxmox on the vpn network, so it look good.
          Like I said, if I do the same ping and traceroute directly via proxmox interface it working good.
          So the problem come from pfSense to send the data to proxmox server (10.1.1.2 to 10.1.1.1) it's like it's only going on one way.

          Doing a traceroute directly via pfSense interface (working one) result on this

          15:18:01.158901 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25985, length 8
15:18:01.168819 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25985, length 8
15:18:01.670255 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25986, length 8
15:18:01.679957 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25986, length 8
15:18:02.188947 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25987, length 8
15:18:02.198911 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25987, length 8
15:18:02.631207 IP 10.23.1.200.47623 > 10.23.1.100.33435: UDP, length 12
15:18:02.641019 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:18:02.641103 IP 10.23.1.200.47623 > 10.23.1.100.33436: UDP, length 12
15:18:02.651024 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:18:02.651124 IP 10.23.1.200.47623 > 10.23.1.100.33437: UDP, length 12
15:18:02.661380 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
15:18:02.661491 IP 10.23.1.200.47623 > 10.23.1.100.33438: UDP, length 12
15:18:02.684359 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33438 unreachable, length 48
15:18:02.684430 IP 10.23.1.200.47623 > 10.23.1.100.33439: UDP, length 12
15:18:02.706763 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33439 unreachable, length 48
15:18:02.706777 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25988, length 8
15:18:02.706821 IP 10.23.1.200.47623 > 10.23.1.100.33440: UDP, length 12
15:18:02.716579 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25988, length 8
15:18:02.729263 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33440 unreachable, length 48
15:18:03.239139 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25989, length 8
15:18:03.248924 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25989, length 8
15:18:03.780349 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25990, length 8
15:18:03.790087 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25990, length 8
15:18:04.321727 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25991, length 8
15:18:04.331676 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25991, length 8
15:18:04.858850 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25992, length 8
15:18:04.868774 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25992, length 8
          stephenw10S 1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by stephenw10

            If Proxmox has an IP in the 10.23.1.0/24 tunnel subnet when you ping anything in the tunnel subnet from it I suspect you are seeing an icmp redirect at some point so the traffic in at least one direction goes directly rather than via pfSense.
            You have a high probability of asymmetric routing there.

            Do you see blocked traffic in the firewall log when it fails?

            Steve

            1 Reply Last reply Reply Quote 0
            • D
              DasK
              last edited by DasK

              Thanks for reply.

              No there is anything blocked, I allow everything into the firewall rules.
              Proxmox don't have an IP in the 10.23.1.0/24 tunnel.

              I just added a rule to tell proxmox to communicate to the 10.23.1.0/24 tunnel via 10.1.1.2(pfSense)

              ip route add 10.23.1.0/24 via 10.1.1.2

              So my proxmox server know where to go to communicate with 10.23.1.0/24 network.
              The thing I don't understand is why ping is working, but not traceroute or rpcinfo, if I do a traceroute -I it work.
              But well, in final I need this rpc working to etablish a NFS storage.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator @DasK
                last edited by

                @DasK said in Route WAN network to OVPN:

                The ip: 10.23.1.200 is the IP of the proxmox on the vpn network

                Then what did you mean by that? A different Proxmox server?

                Steve

                1 Reply Last reply Reply Quote 0
                • D
                  DasK
                  last edited by DasK

                  Ok I'm sorry, I was meaning the IP of the pfSense, my bad

                  Proxmox IP: 10.1.1.1
                  PfSense WAN IP: 10.1.1.2
                  PfSense LAN IP: 192.168.10.254
                  PfSense VPN IP: 10.23.1.200
                  VPN NETWORK: 10.23.1.0/24

                  And the server I try to communicate with on the VPN is : 10.23.1.100

                  Outbound rules (without this, proxmox can't ping 10.23.1.100 anymore)
                  f7e68811-9f28-4dbb-80bb-d4cea4836e97-image.png

                  Interfaces
                  68580d68-37a7-4db9-b423-6d406710b9aa-image.png

                  Wan rules
                  9e837633-8910-412b-b1f1-40908c79675d-image.png

                  Lan rules
                  a846fc18-042b-4962-bb21-b6a28b9afbf7-image.png

                  OPT1 rules
                  921c8484-5d24-4bf6-8bf4-e8064d2f74d2-image.png

                  Routes
                  d2078683-c847-47d7-a22f-c8e1fc198c64-image.png

                  pfSense traceroute:
                  alt text

                  pfSense rpcinfo:
                  alt text

                  Proxmox traceroute:
                  ce7a828b-f205-4f1b-8acf-2ebc68e60065-image.png

                  Proxmox rpcinfo:
                  962e0997-2189-4e34-afb7-4d3a6efd504e-image.png

                  Proxmox ping:
                  29622625-b522-4db6-8ce6-d795089610e4-image.png

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Hmm, you might have to check the OpenVPN server, it's able to reach that at 10.23.1.1 but no further. Something blocking or re-routing it there maybe?

                    Hard to imagine what the difference might be there though. It could only really be TTL....

                    You probably need that outbound NAT rule as those target devices otherwise have no route back.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • D
                      DasK
                      last edited by

                      All if good for the OpenVPN server.

                      Any example of how "outbound NAT rule as those target devices " ?

                      Best

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        The rule you already have for source NATing 10.1.1.0/30 to 10.23.1.200.

                        Devices at 10.23.1.100 (and .1) obviously know how to reach 10.23.1.200 but probably have no route to 10.1.1.0/30. That is the reason you need that rule.

                        Steve

                        1 Reply Last reply Reply Quote 0
                        • D
                          DasK
                          last edited by

                          I added the rule, but still the same problem

                          1 Reply Last reply Reply Quote 0
                          • D
                            DasK
                            last edited by

                            Well I think I'll give up and just also connect the proxmox server to the VPN, so there will be a direct connection to both server.. It just look weird to connect proxmox to VPN, when there is a VM pfSense already connected to this VPN

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              We haven't yet seen a packet capture of a TCP connection failing. That should be revealing. If there is no traffic coming back at all for example.

                              This still looks like some route asymmetry somewhere.

                              Does an icmp traceroute succeed?

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • D
                                DasK
                                last edited by

                                Yes all ICMP traceroute work, like ping.

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  When you have ICMP works but TCP fails it's usually an asymmetric routing problem.

                                  It can also be a packet size issue but that would not normally affect UDP traceroute.

                                  Or it can be a hardware off loading problem if the a NIC/driver somewhere is not doing what it reports it can.

                                  Steve

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    DasK
                                    last edited by

                                    How does I can solve this asymetric routing problem ? Or detect where is located the routing problem.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Run a packet capture showing some TCP connection failing. If you see parts of the TCP setup missing they are either being blocked somewhere or routed some other way.

                                      Steve

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        DasK
                                        last edited by DasK

                                        There is a little diagram, if that can help to understand how is setup

                                        db312123-0f2a-4488-b534-0d7b9062e3c5-image.png

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Yeah, there is some difference between what happens when you run, say, rpcinfo from pfSense directly and when it's run from Proxmox. So I would run a pcap on the OpenVPN interface and compare them.

                                          rpcinfo will create quite a lot of output so you might want to use something simpler like just telneting to a port assuming that also fails from Proxmox.

                                          Steve

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.