Site2Site does not work/route in both directions



  • Hi!

    I do have a strange problem with OpenVPN routing and I am pretty sure it was working some time ago.

    It is a site to site link conecting 172.27.72.0/24 (remote) with 10.252.200.0/22. 10.252.200.254 is the OpenVPN server on pfSense, 172.27.72.254 is the client side pfSense. OpenVPN tunnel network is 172.30.14.0/24 (172.30.14.1=server, 172.30.14.2=client).

    From remote (172.27.72.x) I can ping and reach the full 10.252.20x.y network.
    But I cannot reach the remote devices from the server side's network. I can only ping the remote pfSense but no other device.

    What I get is this from server's network - ping:
    PING 172.27.72.1 (172.27.72.1) 56(84) bytes of data.
    From 172.30.14.1 icmp_seq=1 Time to live exceeded
    From 172.30.14.1 icmp_seq=2 Time to live exceeded
    From 172.30.14.1 icmp_seq=3 Time to live exceeded
    From 172.30.14.1 icmp_seq=4 Time to live exceeded
    ...

    What I get is this from server's network - traceroute:
    traceroute to 172.27.72.1 (172.27.72.1), 30 hops max, 60 byte packets
    1 10.252.200.254 0.338 ms 0.362 ms 0.356 ms
    2 172.30.14.1 27.970 ms 35.843 ms 42.439 ms
    3 172.30.14.1 70.223 ms 70.251 ms 74.688 ms
    ...
    29 172.30.14.1 881.594 ms 871.410 ms 885.545 ms
    30 172.30.14.1 907.684 ms 895.962 ms 896.176 ms

    This traceroute is really strange.

    ssh from pfSense server to a device on the client network says:
    ssh -l user 172.27.72.1
    ssh: connect to host 172.27.72.1 port 22: No route to host

    But I see the route, below (***).

    The routing table on the server looks like this (two WANs and two additional OpenVPN servers):
    Destination Gateway Flags Netif Expire
    default 24.134.xx.yyy UGS igb1
    1.0.0.1 82.82.xx.yyy UGHS pppoe0
    1.1.1.1 24.134.xx.yyy UGHS igb1
    4.2.2.1 24.134.xx.yyy UGHS igb1
    4.2.2.2 82.82.xx.yyy UGHS pppoe0
    4.2.2.3 24.134.xx.yyy UGHS igb1
    4.2.2.4 pppoe0 UHS pppoe0
    8.8.4.4 82.82.xx.yyy UGHS pppoe0
    8.8.8.8 24.134.xx.yyy UGHS igb1
    9.9.9.10 24.134.xx.yyy UGHS igb1
    10.252.200.0/22 link#3 U igb2
    10.252.200.254 link#3 UHS lo0
    10.254.154.54 link#3 UHS lo0
    10.254.154.54/32 link#3 U igb2
    24.134.xx.yyy/30 link#2 U igb1
    24.134.xx.yyy link#2 UHS lo0
    82.82.xx.yyy link#12 UH pppoe0
    127.0.0.1 link#6 UH lo0
    172.25.254.0/24 link#11 U igb2.222
    172.25.254.254 link#11 UHS lo0
    172.27.72.0/24 172.30.14.2 UGS ovpns6 (***)
    172.30.1.0/24 172.30.1.2 UGS ovpns1
    172.30.1.1 link#17 UHS lo0
    172.30.1.2 link#17 UH ovpns1
    172.30.2.0/24 172.30.2.2 UGS ovpns3
    172.30.2.1 link#16 UHS lo0
    172.30.2.2 link#16 UH ovpns3
    172.30.14.1 link#14 UHS lo0
    172.30.14.2 link#14 UH ovpns6
    172.30.15.1 link#15 UHS lo0
    172.30.15.2 link#15 UH ovpns7
    176.95.xxx.yyy link#12 UHS lo0
    192.168.55.0/24 link#10 U igb2.55
    192.168.55.254 link#10 UHS lo0

    On the client I have the following routing table (two additional OpenVPN servers):
    Destination Gateway Flags Netif Expire
    default 90.187.xxx.yyy UGS igb0
    10.252.200.0/22 172.30.14.1 UGS ovpnc1
    90.187.xxx.yyy/30 link#1 U igb0
    90.187.xxx.yyy link#1 UHS lo0
    127.0.0.1 link#5 UH lo0
    172.27.72.0/24 link#3 U igb2
    172.27.72.254 link#3 UHS lo0
    172.30.14.1 link#11 UH ovpnc1
    172.30.14.2 link#11 UHS lo0
    172.30.15.1 link#12 UH ovpnc2
    172.30.15.2 link#12 UHS lo0
    172.30.50.0/24 172.30.50.2 UGS ovpns3
    172.30.50.1 link#9 UHS lo0
    172.30.50.2 link#9 UH ovpns3
    172.30.51.0/24 172.30.51.2 UGS ovpns4
    172.30.51.1 link#10 UHS lo0
    172.30.51.2 link#10 UH ovpns4
    192.168.55.0/24 link#8 U igb2.55
    192.168.55.254 link#8 UHS lo0

    The server config looks like this:
    dev ovpns6
    verb 1
    dev-type tun
    dev-node /dev/tun6
    writepid /var/run/openvpn_server6.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 24.134.xxx.yyy
    engine cryptodev
    ifconfig 172.30.14.1 172.30.14.2
    lport 1196
    management /var/etc/openvpn/server6.sock unix
    max-clients 16
    route 172.27.72.0 255.255.255.0
    secret /var/etc/openvpn/server6.secret
    fast-io
    sndbuf 1048576
    rcvbuf 1048576

    The client config looks like this:
    dev ovpnc1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 90.187.xxx.yyy
    engine cryptodev
    lport 0
    management /var/etc/openvpn/client1.sock unix
    remote server 1196
    ifconfig 172.30.14.2 172.30.14.1
    route 10.252.200.0 255.255.252.0
    secret /var/etc/openvpn/client1.secret
    resolv-retry infinite
    fast-io
    sndbuf 1048576
    rcvbuf 1048576

    On the client I have some sort of fail-over using a gateway group. See https://forum.netgate.com/topic/147314/fail-over-client-config/9

    Can someone please have a look and help me find what's wrong? What am I missing?
    I need to reach all devices from both sides of the VPN.

    Thanks a lot!


  • LAYER 8

    both side are on the same subnet ?

    it seems the packet are bounced between the hops

    3 172.30.14.1 70.223 ms 70.251 ms 74.688 ms
    ...
    29 172.30.14.1 881.594 ms 871.410 ms 885.545 ms
    30 172.30.14.1 907.684 ms 895.962 ms 896.176 ms

    it's a routing problem



  • No, they are not on the same subnet.
    Server is on 10.252.200.0/22, client is on 172.27.72.0/24.
    Server pfSense has 10.252.200.254, client pfSense has 172.27.72.254.
    Both share the same tunnel network (172.30.14.0/24) with server pfSense at 172.30.14.1 and client pfSense on 172.30.14.2.
    Yes, it seems to be a routing problem. But I cannot see where. The routing table looks good to me.
    From client pfSense I can ping the whole 172.27.72.0/24 network.
    From client network I can ping the whole 10.252.200.0/22 network via VPN.
    From server pfSense network I can ping nothing on the client network, except client pfSense on client network. Not even the server pfSense can ping anything else than the client pfSense.
    And I don't see why!

    I just checked, the firewalls on both ends do not block anything.


  • LAYER 8

    @demux said in Site2Site does not work/route in both directions:
    client:
    172.27.72.0/24 link#3 U igb2
    172.27.72.254 link#3 UHS lo0
    172.30.14.1 link#11 UH ovpnc1
    172.30.14.2 link#11 UHS lo0
    10.252.200.0/22 172.30.14.1 UGS ovpnc1

    server:
    10.252.200.0/22 link#3 U igb2
    10.252.200.254 link#3 UHS lo0
    172.30.14.1 link#14 UHS lo0
    172.30.14.2 link#14 UH ovpns6
    172.27.72.0/24 172.30.14.2 UGS ovpns6

    from the client side it work but not from the server side
    šŸ¤”



  • Correct.


  • LAYER 8

    when did it stop working? after an update maybe?
    a reboot didn't help i suppose?
    i can't see what's wrong ...
    can you make a screenshot of the firewall rules ?
    do you have any floating rules?



  • I did reboot. Did not update.
    I am not sure if it ever worked. I believe, but I'm not sure.
    Maybe it stopped working after that gateway fail-over thing mentioned in the link. I did reverse that temporarily but did not help.
    To me everything looks fine. And why the problems into this direction, not the other or both.
    I have real problems testing that, because the distance between them is about 800km. If something breaks I have to travel about 9 hours. Like last weekend...



  • I have no clue what happened.
    I deleted all servers and clients and started over again.
    I strictly followed this documentation:
    https://docs.netgate.com/pfsense/en/latest/book/openvpn/openvpn-and-multi-wan.html#
    Server uses GW group and client uses remote ...
    And now it works.
    Whatever happened to my previous config.


  • LAYER 8

    the routing table now is the same ?
    maybe it was something else on the configuration