Packets disappear between vpn client and vpn server



  • I have a hub and spoke site-to-site vpn setup, however – only in one direction and only on one spoke -- nodes on the client lan  cannot reach nodes on the server lan, although they can reach the server itself.  The opposite direction works fine (i.e. nodes on server lan can reach nodes on this client-lan.

    The really odd thing is that the problem is not between the vpn server and the node on its lan.  I know this because when I attempt to reach the server from the client-node, I can see the packets reach the server in the tcp dump, however if I try to get beyond the server to one of the nodes on the server-lan, the packets appear to vanish on the vpn client, never making it to the server.

    Here is a diagram I made of the situation:

    
       A ====> B ====> C ====> D  ssh from node on server lan to node on client lan works, as does any portion of that
       A       B <==== C <==== D  ssh from node on client lan to vpn server works
       A <==== B <==== C       D  ssh from vpn client to node on server lan works
       A XXXXX B XXXXX C <==== D  ssh from node on client lan to node on server lan fails!!!  packets disappear between ovpnc and ovpns interfaces
    
    Server    VPN     VPN    Client
     LAN     Server  Client   LAN
    
    

    While it would seem this has to be a firewall / NAT problem, I've been over my rules time and again, as well as logs (no blocks or rejections), and I've pfctl -d on both vpn server and client.

    Note that another VPN client lan and vpn client node exists, E and F, which work fine in both directions.

    Here is the routing table on the vpn server:

    
    : netstat -r
    Routing tables
    
    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            65.145.227.209     UGS      vtnet0
    10.0.0.0/24        link#7             U       bridge0
    10.0.0.1           link#7             UHS         lo0
    10.5.0.0/24        10.0.0.100         UGS     bridge0
    10.10.0.0/24       10.0.0.101         UGS     bridge0
    10.20.0.0/24       10.0.0.102         UGS     bridge0
    63.141.227.208/29  link#1             U        vtnet0
    sangat.fortulite.n link#1             UHS         lo0
    localhost          link#4             UH          lo0
    
    

    … and on the vpn client ...

    
    : netstat -r
    Routing tables
    
    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            lo0-100.NWRKNJ-VFT UGS         em1
    10.0.0.0/24        link#8             U        ovpnc1
    10.0.0.102         link#8             UHS         lo0
    10.5.0.0/24        10.0.0.100         UGS      ovpnc1
    10.10.0.0/24       10.0.0.101         UGS      ovpnc1
    10.20.0.0/24       link#1             U           em0
    pfSense            link#1             UHS         lo0
    10.250.0.0/24      10.255.255.1       UGS      ovpnc2
    10.250.0.0/16      10.255.255.1       UGS      ovpnc2
    10.255.255.0/24    10.255.255.1       UGS      ovpnc2
    10.255.255.1       link#9             UH       ovpnc2
    10.255.255.4       link#9             UHS         lo0
    nsphil01.verizon.n d6:1b:af:2b:e3:98  UHS         em1
    nsnwrk01.verizon.n d6:1b:af:2b:e3:98  UHS         em1
    93.232.81.0/24     link#2             U           em1
    pool-96-242-81-51\. link#2             UHS         lo0
    localhost          link#3             UH          lo0
    
    

    for reference, the is a vpn-client that does not have the problem:

    
     netstat -r
    Routing tables
    
    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            lo0-100.NWRKNJ-VFT UGS         em1
    10.0.0.0/24        link#8             U        ovpnc1
    10.0.0.100         link#8             UHS         lo0
    10.5.0.0/24        link#1             U           em0
    ads-120elmst-pfsen link#1             UHS         lo0
    10.10.0.0/24       10.0.0.101         UGS      ovpnc1
    10.20.0.0/24       10.0.0.102         UGS      ovpnc1
    10.30.0.0/24       10.0.0.103         UGS      ovpnc1
    10.250.0.0/24      10.255.255.1       UGS      ovpnc2
    10.250.0.0/16      10.255.255.1       UGS      ovpnc2
    10.255.255.0/24    10.255.255.1       UGS      ovpnc2
    10.255.255.1       link#9             UH       ovpnc2
    10.255.255.2       link#9             UHS         lo0
    nsphil01.verizon.n 48:5d:36:85:1e:48  UHS         em1
    nsnwrk01.verizon.n 48:5d:36:85:1e:48  UHS         em1
    93.242.143.0/24    link#2             U           em1
    pool-96-242-148-23 link#2             UHS         lo0
    localhost          link#3             UH          lo0
    
    

    The node on the client lan is set up to use the vpn-client as router/gateway:

    
    # ip route list
    default via 10.20.0.1 dev vmbr0
    10.20.0.0/24 dev vmbr0  proto kernel  scope link  src 10.20.0.251
    
    

    However, the node on the server lan has it's own wan, and also the routes must be statically set up (no way to push from client to server, afaik):

    
    # ip route list
    default via 63.141.227.209 dev vmbr0 onlink
    10.0.0.0/24 dev vmbr1 proto kernel scope link src 10.0.0.250
    10.5.0.0/24 via 10.0.0.1 dev vmbr1
    10.10.0.0/24 via 10.0.0.1 dev vmbr1
    10.20.0.0/24 via 10.0.0.1 dev vmbr1
    66.141.226.208/29 dev vmbr0 proto kernel scope link src 63.141.227.210
    
    

    I'm starting to lean toward a bug somewhere, rather than a mistake in my implementation.  I'm at a loss, and any help is appreciated.


Log in to reply