Route WAN network to OVPN



  • Hello,

    I have a proxmox cloud server with a VM pfSense.

    Proxmox server: 10.1.1.1/30
    PfSense WAN: 10.1.1.2/30
    PfSense LAN: 192.168.10.0/24

    The pfSense is connected to a remote VPN server (on another pfSense server)
    VPN: 10.23.1.1/24

    I added an outbound rule to redirect WAN to VPN interface (if I'm not wrong)
    f33e1a83-0de3-4e2e-b507-350d24d568bf-image.png

    Now from the pfSense itself, I'm able to ping/traceroute/rpcinfo a server I have inside the VPN tunnel from the WAN interface, IP: 10.23.1.100
    5f70ecf6-93d7-4431-bfcb-b22a273f3e4d-image.png
    62fe640f-abc2-4db2-89d8-83efe8273af9-image.png

    Now, I switch to proxmox server, I added the route to the VPN

    Now the strange thing start, I'm able to ping the remote server (10.23.1.100), but not able to fully traceroute it or rpcinfo

    d9d062de-e0bd-4a94-ab4b-bf2370703c84-image.png

    The iptables rules are correct (tried with full allow rules)
    Inside pfSense, I allow traffic from * to * on all interfaces.

    It seems like my server can communicate to it, but I don't have the answer but the strange thing is that the pfSense WAN 10.1.1.2 can do it, but not my proxmox server 10.1.1.1 ..

    Thanks for your help


  • Netgate Administrator

    I assume OPT1 is the assigned OpenVPN interface?

    Check the state table when you are trying to connect or successfully pinging. Is the traffic NAT'd to the OPT1 interface IP as expected?

    Run a packet capture on the OpenVPN interface. What actually being sent from there?

    Steve



  • I done the Packet Capture on the OPT1 interface while doing a ping and a traceroute on the proxmox serveur

    15:09:18.758929 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 24999, length 8
    15:09:18.768983 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 24999, length 8
    15:09:19.289009 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25000, length 8
    15:09:19.299123 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25000, length 8
    15:09:19.830176 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25001, length 8
    15:09:19.840291 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25001, length 8
    15:09:20.371708 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25002, length 8
    15:09:20.381646 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25002, length 8
    15:09:20.888891 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25003, length 8
    15:09:20.898926 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25003, length 8
    15:09:21.418876 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 1, length 64
    15:09:21.419019 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25004, length 8
    15:09:21.428766 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25004, length 8
    15:09:21.441468 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 1, length 64
    15:09:21.937154 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25005, length 8
    15:09:21.946987 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25005, length 8
    15:09:22.420103 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 2, length 64
    15:09:22.442994 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 2, length 64
    15:09:22.448187 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25006, length 8
    15:09:22.458130 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25006, length 8
    15:09:22.955888 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25007, length 8
    15:09:22.965827 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25007, length 8
    15:09:23.420449 IP 10.23.1.200 > 10.23.1.100: ICMP echo request, id 16951, seq 3, length 64
    15:09:23.443246 IP 10.23.1.100 > 10.23.1.200: ICMP echo reply, id 16951, seq 3, length 64
    15:09:23.489014 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25008, length 8
    15:09:23.498938 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25008, length 8
    15:09:24.030321 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25009, length 8
    15:09:24.040293 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25009, length 8
    15:09:24.545142 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25010, length 8
    15:09:24.555173 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25010, length 8
    15:09:25.058926 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25011, length 8
    15:09:25.068957 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25011, length 8
    15:09:25.537847 IP 10.23.1.200.35330 > 10.23.1.100.33437: UDP, length 32
    15:09:25.537863 IP 10.23.1.200.18025 > 10.23.1.100.33438: UDP, length 32
    15:09:25.537871 IP 10.23.1.200.2270 > 10.23.1.100.33439: UDP, length 32
    15:09:25.537879 IP 10.23.1.200.23400 > 10.23.1.100.33440: UDP, length 32
    15:09:25.537886 IP 10.23.1.200.39105 > 10.23.1.100.33441: UDP, length 32
    15:09:25.537894 IP 10.23.1.200.14561 > 10.23.1.100.33442: UDP, length 32
    15:09:25.537917 IP 10.23.1.200.5445 > 10.23.1.100.33443: UDP, length 32
    15:09:25.537925 IP 10.23.1.200.14700 > 10.23.1.100.33444: UDP, length 32
    15:09:25.537933 IP 10.23.1.200.1751 > 10.23.1.100.33445: UDP, length 32
    15:09:25.537954 IP 10.23.1.200.49371 > 10.23.1.100.33446: UDP, length 32
    15:09:25.537961 IP 10.23.1.200.35188 > 10.23.1.100.33447: UDP, length 32
    15:09:25.537969 IP 10.23.1.200.18223 > 10.23.1.100.33448: UDP, length 32
    15:09:25.537992 IP 10.23.1.200.56560 > 10.23.1.100.33449: UDP, length 32
    15:09:25.540117 IP 10.23.1.200.14513 > 10.23.1.100.33450: UDP, length 32
    15:09:25.540135 IP 10.23.1.200.34198 > 10.23.1.100.33451: UDP, length 32
    15:09:25.540147 IP 10.23.1.200.40572 > 10.23.1.100.33452: UDP, length 32
    15:09:25.548161 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:09:25.548208 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:09:25.548233 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:09:25.550430 IP 10.23.1.200.44090 > 10.23.1.100.33453: UDP, length 32
    15:09:25.550444 IP 10.23.1.200.30073 > 10.23.1.100.33454: UDP, length 32
    15:09:25.550453 IP 10.23.1.200.46325 > 10.23.1.100.33455: UDP, length 32
    15:09:25.588888 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25012, length 8
    15:09:25.598760 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25012, length 8
    15:09:26.130202 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25013, length 8
    15:09:26.140134 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25013, length 8
    15:09:26.658825 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25014, length 8
    15:09:26.668659 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25014, length 8
    15:09:27.180239 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25015, length 8
    15:09:27.190104 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25015, length 8
    15:09:27.705401 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25016, length 8
    15:09:27.715239 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25016, length 8
    15:09:28.220925 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25017, length 8
    15:09:28.231023 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25017, length 8
    15:09:28.754789 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25018, length 8
    15:09:28.764623 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25018, length 8
    15:09:29.273490 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25019, length 8
    

    The ip: 10.23.1.200 is the IP of the proxmox on the vpn network, so it look good.
    Like I said, if I do the same ping and traceroute directly via proxmox interface it working good.
    So the problem come from pfSense to send the data to proxmox server (10.1.1.2 to 10.1.1.1) it's like it's only going on one way.

    Doing a traceroute directly via pfSense interface (working one) result on this

    15:18:01.158901 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25985, length 8
    15:18:01.168819 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25985, length 8
    15:18:01.670255 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25986, length 8
    15:18:01.679957 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25986, length 8
    15:18:02.188947 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25987, length 8
    15:18:02.198911 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25987, length 8
    15:18:02.631207 IP 10.23.1.200.47623 > 10.23.1.100.33435: UDP, length 12
    15:18:02.641019 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:18:02.641103 IP 10.23.1.200.47623 > 10.23.1.100.33436: UDP, length 12
    15:18:02.651024 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:18:02.651124 IP 10.23.1.200.47623 > 10.23.1.100.33437: UDP, length 12
    15:18:02.661380 IP 10.23.1.1 > 10.23.1.200: ICMP time exceeded in-transit, length 36
    15:18:02.661491 IP 10.23.1.200.47623 > 10.23.1.100.33438: UDP, length 12
    15:18:02.684359 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33438 unreachable, length 48
    15:18:02.684430 IP 10.23.1.200.47623 > 10.23.1.100.33439: UDP, length 12
    15:18:02.706763 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33439 unreachable, length 48
    15:18:02.706777 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25988, length 8
    15:18:02.706821 IP 10.23.1.200.47623 > 10.23.1.100.33440: UDP, length 12
    15:18:02.716579 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25988, length 8
    15:18:02.729263 IP 10.23.1.100 > 10.23.1.200: ICMP 10.23.1.100 udp port 33440 unreachable, length 48
    15:18:03.239139 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25989, length 8
    15:18:03.248924 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25989, length 8
    15:18:03.780349 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25990, length 8
    15:18:03.790087 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25990, length 8
    15:18:04.321727 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25991, length 8
    15:18:04.331676 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25991, length 8
    15:18:04.858850 IP 10.23.1.200 > 10.23.1.1: ICMP echo request, id 10458, seq 25992, length 8
    15:18:04.868774 IP 10.23.1.1 > 10.23.1.200: ICMP echo reply, id 10458, seq 25992, length 8
    

  • Netgate Administrator

    If Proxmox has an IP in the 10.23.1.0/24 tunnel subnet when you ping anything in the tunnel subnet from it I suspect you are seeing an icmp redirect at some point so the traffic in at least one direction goes directly rather than via pfSense.
    You have a high probability of asymmetric routing there.

    Do you see blocked traffic in the firewall log when it fails?

    Steve



  • Thanks for reply.

    No there is anything blocked, I allow everything into the firewall rules.
    Proxmox don't have an IP in the 10.23.1.0/24 tunnel.

    I just added a rule to tell proxmox to communicate to the 10.23.1.0/24 tunnel via 10.1.1.2(pfSense)

    ip route add 10.23.1.0/24 via 10.1.1.2
    

    So my proxmox server know where to go to communicate with 10.23.1.0/24 network.
    The thing I don't understand is why ping is working, but not traceroute or rpcinfo, if I do a traceroute -I it work.
    But well, in final I need this rpc working to etablish a NFS storage.


  • Netgate Administrator

    @DasK said in Route WAN network to OVPN:

    The ip: 10.23.1.200 is the IP of the proxmox on the vpn network

    Then what did you mean by that? A different Proxmox server?

    Steve



  • Ok I'm sorry, I was meaning the IP of the pfSense, my bad

    Proxmox IP: 10.1.1.1
    PfSense WAN IP: 10.1.1.2
    PfSense LAN IP: 192.168.10.254
    PfSense VPN IP: 10.23.1.200
    VPN NETWORK: 10.23.1.0/24

    And the server I try to communicate with on the VPN is : 10.23.1.100

    Outbound rules (without this, proxmox can't ping 10.23.1.100 anymore)
    f7e68811-9f28-4dbb-80bb-d4cea4836e97-image.png

    Interfaces
    68580d68-37a7-4db9-b423-6d406710b9aa-image.png

    Wan rules
    9e837633-8910-412b-b1f1-40908c79675d-image.png

    Lan rules
    a846fc18-042b-4962-bb21-b6a28b9afbf7-image.png

    OPT1 rules
    921c8484-5d24-4bf6-8bf4-e8064d2f74d2-image.png

    Routes
    d2078683-c847-47d7-a22f-c8e1fc198c64-image.png

    pfSense traceroute:
    alt text

    pfSense rpcinfo:
    alt text

    Proxmox traceroute:
    ce7a828b-f205-4f1b-8acf-2ebc68e60065-image.png

    Proxmox rpcinfo:
    962e0997-2189-4e34-afb7-4d3a6efd504e-image.png

    Proxmox ping:
    29622625-b522-4db6-8ce6-d795089610e4-image.png


  • Netgate Administrator

    Hmm, you might have to check the OpenVPN server, it's able to reach that at 10.23.1.1 but no further. Something blocking or re-routing it there maybe?

    Hard to imagine what the difference might be there though. It could only really be TTL....

    You probably need that outbound NAT rule as those target devices otherwise have no route back.

    Steve



  • All if good for the OpenVPN server.

    Any example of how "outbound NAT rule as those target devices " ?

    Best


  • Netgate Administrator

    The rule you already have for source NATing 10.1.1.0/30 to 10.23.1.200.

    Devices at 10.23.1.100 (and .1) obviously know how to reach 10.23.1.200 but probably have no route to 10.1.1.0/30. That is the reason you need that rule.

    Steve



  • I added the rule, but still the same problem



  • Well I think I'll give up and just also connect the proxmox server to the VPN, so there will be a direct connection to both server.. It just look weird to connect proxmox to VPN, when there is a VM pfSense already connected to this VPN


  • Netgate Administrator

    We haven't yet seen a packet capture of a TCP connection failing. That should be revealing. If there is no traffic coming back at all for example.

    This still looks like some route asymmetry somewhere.

    Does an icmp traceroute succeed?

    Steve



  • Yes all ICMP traceroute work, like ping.


  • Netgate Administrator

    When you have ICMP works but TCP fails it's usually an asymmetric routing problem.

    It can also be a packet size issue but that would not normally affect UDP traceroute.

    Or it can be a hardware off loading problem if the a NIC/driver somewhere is not doing what it reports it can.

    Steve



  • How does I can solve this asymetric routing problem ? Or detect where is located the routing problem.


  • Netgate Administrator

    Run a packet capture showing some TCP connection failing. If you see parts of the TCP setup missing they are either being blocked somewhere or routed some other way.

    Steve



  • There is a little diagram, if that can help to understand how is setup

    db312123-0f2a-4488-b534-0d7b9062e3c5-image.png


  • Netgate Administrator

    Yeah, there is some difference between what happens when you run, say, rpcinfo from pfSense directly and when it's run from Proxmox. So I would run a pcap on the OpenVPN interface and compare them.

    rpcinfo will create quite a lot of output so you might want to use something simpler like just telneting to a port assuming that also fails from Proxmox.

    Steve


Log in to reply