OpenVPN site-2-site - VLAN host cannot receive response from the other end



  • Hi,

    I chose this section although I must admit I'm not sure it's a VLAN-related issue.

    I'm currently setting up a new network in parallel to an old one and to have a connection between these two during migration phase, I established an OpenVPN site-2-site tunnel.

    I really hope I'm not making a fool of myself because I missed something obvious here. Anyway, I'd be happy to get this working. :)

    This figure illustrates the setup:
    0_1534846352834_site-2-site.png

    What works

    • the pfSense from network A can ping Linux host B in network B via site-2-site tunnel
    • from the pfSense I can initiate a SSH connection to Linux host B

    What doesn't work

    • Linux host A from network A cannot reach Linux host B in network B or to be more precise, the ICMP packet gets through to Linux host B but the reply doesn't find the way back

    What I already figured out

    • connections from the pfSense to Linux host B have source IP 10.42.42.2 (the OpenVPN endpoint)
    • connections from Linux host A to Linux host B have source IP 192.168.101.3
    • the result is that response packets back to the pfSense go through the tunnel while response attempts to Linux host A go through the non-tunnel network (and that's not working and expected)

    Now you might say it's a routing issue, but what I don't understand is, why does the pfSense behave differently depending on the source of the packets? I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN? Do I need to manually create routes per VLAN to make it go through the tunnel?

    Some pieces of information that might be relevant

    Ping attempt from Linux host A to Linux host B sniffed on Linux host B (the reply never reaches Linux host A)

    14:22:58.924614 IP 192.168.101.3 > 192.168.189.101: ICMP echo request, id 20267, seq 1, length 64
    14:22:58.924640 IP 192.168.189.101 > 192.168.101.3: ICMP echo reply, id 20267, seq 1, length 64
    

    Ping attempt from pfSense to Linux host B sniffed on Linux host B (works fine)

    14:23:20.770721 IP 10.42.42.2 > 192.168.189.101: ICMP echo request, id 58389, seq 0, length 64
    14:23:20.770749 IP 192.168.189.101 > 10.42.42.2: ICMP echo reply, id 58389, seq 0, length 64
    

    pfSense OpenVPN config

    dev ovpnc1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-256-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 192.168.178.28
    engine cryptodev
    lport 51195
    management /var/etc/openvpn/client1.sock unix
    remote 192.168.178.13 51195
    ifconfig 10.42.42.2 10.42.42.1
    route 192.168.181.0 255.255.255.0
    route 192.168.189.0 255.255.255.0
    route 192.168.190.0 255.255.255.0
    secret /var/etc/openvpn/client1.secret 
    resolv-retry infinite
    

    Route check on pfSense

    route get 192.168.189.101
       route to: 192.168.189.101
    destination: 192.168.189.0
           mask: 255.255.255.0
        gateway: 10.42.42.1
            fib: 0
      interface: ovpnc1
          flags: <UP,GATEWAY,DONE,STATIC>
     recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
           0         0         0         0      1500         1         0 
    

    Route check on Linux host A (where 192.168.101.1 is the pfSense)

    ip r get 192.168.189.101
    192.168.189.101 via 192.168.101.1 dev enp3s4f0 src 192.168.101.3 
        cache
    

    pfSense interfaces

    lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=e500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    	ether 00:08:a2:0d:62:ff
    	inet6 fe80::208:a2ff:fe0d:62ff%lagg0 prefixlen 64 scopeid 0x13 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet autoselect
    	status: active
    	groups: lagg 
    	laggproto loadbalance lagghash l2,l3,l4
    	laggport: ix2 flags=4<ACTIVE>
    	laggport: ix3 flags=4<ACTIVE>
    lagg0.101: flags=88943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
    	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    	ether 00:08:a2:0d:62:ff
    	inet6 fe80::208:a2ff:fe0d:62ff%lagg0.101 prefixlen 64 scopeid 0x17 
    	inet 192.168.101.1 netmask 0xffffff00 broadcast 192.168.101.255 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet autoselect
    	status: active
    	vlan: 101 vlanpcp: 0 parent interface: lagg0
    	groups: vlan 
    ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
    	options=80000<LINKSTATE>
    	inet6 fe80::208:a2ff:fe0d:62fd%ovpnc1 prefixlen 64 scopeid 0x18 
    	inet 10.42.42.2 --> 10.42.42.1  netmask 0xffffffff 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	groups: tun openvpn 
    	Opened by PID 77923
    


  • @tumble said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:

    I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN?

    Why do you think, it isn't?
    Try a packet capture on the pfSense or the Edge routers VPN interface to be shure.

    I guess, the routing problem is on the Edge. So check the routes there.



  • @viragomann said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:

    I guess, the routing problem is on the Edge. So check the routes there.

    Thanks for your reply, I think you are right. Debugging all possible things on different hops for hours got me confused and especially the different source IPs depending on whether the packet originates from the pfSense itself or from a VLAN behind it caught more of my attention than it should, I guess.

    The packets flow through the ovpnc1 interface, from the pfSense itself and from the VLAN as well, the only difference being the source IP but I assume that's normal.

    My EdgeRouter routing table looks like

    root@edgerouter:~# ip r
    0.0.0.0/24 dev vtun1  proto kernel  scope link 
    default via 192.168.178.1 dev eth1  proto zebra 
    10.0.142.1 dev vtun2  proto kernel  scope link 
    10.0.142.2 dev vtun2  proto kernel  scope link  src 10.0.142.1 
    10.6.66.0/24 dev vtun3  proto kernel  scope link  src 10.6.66.1 
    10.7.0.1 dev vtun0  proto kernel  scope link  src 10.7.0.2 
    10.7.0.2 dev vtun0  proto kernel  scope link 
    10.9.0.0/24 dev vtun0  proto zebra 
    10.10.0.0/24 dev vtun1  proto kernel  scope link  src 10.10.0.1 
    10.15.0.0/24 dev vtun0  proto zebra 
    10.42.42.1 dev vtun4  proto kernel  scope link 
    10.42.42.2 dev vtun4  proto kernel  scope link  src 10.42.42.1 
    192.168.10.0/24 dev eth2  proto kernel  scope link  src 192.168.10.10 
    192.168.101.0/24 dev vtun4  proto zebra 
    192.168.142.0/24 dev vtun2  proto zebra 
    192.168.143.0/24 dev vtun2  proto zebra 
    192.168.178.0/24 dev eth1  proto kernel  scope link  src 192.168.178.13 
    192.168.180.0/24 dev vtun0  proto zebra 
    192.168.181.0/24 dev eth3  proto kernel  scope link  src 192.168.181.1 
    192.168.188.0/24 dev vtun0  proto zebra 
    192.168.189.0/24 dev eth4  proto kernel  scope link  src 192.168.189.1 
    192.168.190.0/24 dev eth6  proto kernel  scope link  src 192.168.190.1 
    192.168.191.0/24 dev vtun0  proto zebra 
    192.168.199.0/24 dev eth0  proto kernel  scope link  src 192.168.199.1 
    192.168.200.0/24 dev eth5  proto kernel  scope link  src 192.168.200.1 
    

    and I would expect the response packet to 192.168.101.3 to go via vtun4 but according to this output

    root@edgerouter:~# ip r get 192.168.101.3
    192.168.101.3 via 192.168.178.28 dev eth1  src 192.168.189.1 
        cache
    

    it doesn't. Takes the way via WAN iface. Didn't figure out yet how I can flush this cache to make sure it's not a caching issue, but I guess I will do some more research in that direction for now, unless someone here finds some conceptional mistake in my setup.



  • I'm wondering why there is no gateway mentioned in the route for 192.168.101.0/24. 10.42.42.2 should by the gateway to route the traffic to.

    Is the OpenVPN configured as a site-to-site?



  • I'm struggling quite a bit with understanding the EdgeRouter, which is one of the reasons I'm migrating everything to pfSense which feels much more intuitive to me.

    There are two ways to set a route on the EdgeRouter - I can create a static route via UI (or CLI) which results in the entries you can see above and that basically works. I know that, because I have 2 more OpenVPN site-2-site tunnels, one to another EdgeRouter in a second office and one to a parallel in-house infrastructure behind another pfSense. The only difference between my 2 pfSenses, as far as I can tell, is that the new one uses VLANs (and has this fancy and a little confusing new switch, but I guess that's not involved here). Guess that's also what made me post in this section, because I'm relatively new to using them and thought I'm missing something here.
    The other way of setting a route is by passing a parameter in the OpenVPN config (which is pretty much what pfSense seems to be doing). The route looks different in the table then, it's proto kernel instead of zebra. I tried that once while debugging but didn't have the feeling it solved anything. I might go that way once more and check if the route get shows a different result.

    I'll show you the OpenVPN config on EdgeRouter side anyway, maybe you see something weird

     description "pfsense-redacted uplink"
     encryption aes256
     firewall {
         in {
             name PFSENSE-REDACTED-VPN
         }
         local {
             name PFSENSE-REDACTED-VPN
         }
         out {
             name PFSENSE-REDACTED-VPN
         }
     }
     local-address 10.42.42.1 {
     }
     local-port 51195
     mode site-to-site
     openvpn-option --float
     openvpn-option "--ping 10"
     openvpn-option "--ping-restart 20"
     openvpn-option --ping-timer-rem
     openvpn-option --persist-tun
     openvpn-option --persist-key
     openvpn-option "--user nobody"
     openvpn-option "--group nogroup"
     openvpn-option ""
     openvpn-option "--verb 4"
     remote-address 10.42.42.2
     remote-host 192.168.178.28
     remote-port 51195
     shared-secret-key-file /config/auth/pfsense-redacted/secret
    

  • Netgate

    That's all great but this is not edgerouter support.

    It appears the pfSense side is fine but the edgerouter is not routing traffic for 192.168.101.0/24 back over the tunnel.

    That said, try adding an OpenVPN option on the edgerouter that results in this:

    "--route 192.168.101.0 255.255.255.0"

    edit -

    Probably not since the zebra route is in the table to the correct tunnel it must be getting that from somewhere else. Probably have to ask them.