OpenVPN site-2-site - VLAN host cannot receive response from the other end

tumble

Hi,

I chose this section although I must admit I'm not sure it's a VLAN-related issue.

I'm currently setting up a new network in parallel to an old one and to have a connection between these two during migration phase, I established an OpenVPN site-2-site tunnel.

I really hope I'm not making a fool of myself because I missed something obvious here. Anyway, I'd be happy to get this working. :)

This figure illustrates the setup:

What works

the pfSense from network A can ping Linux host B in network B via site-2-site tunnel
from the pfSense I can initiate a SSH connection to Linux host B

What doesn't work

Linux host A from network A cannot reach Linux host B in network B or to be more precise, the ICMP packet gets through to Linux host B but the reply doesn't find the way back

What I already figured out

connections from the pfSense to Linux host B have source IP 10.42.42.2 (the OpenVPN endpoint)
connections from Linux host A to Linux host B have source IP 192.168.101.3
the result is that response packets back to the pfSense go through the tunnel while response attempts to Linux host A go through the non-tunnel network (and that's not working and expected)

Now you might say it's a routing issue, but what I don't understand is, why does the pfSense behave differently depending on the source of the packets? I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN? Do I need to manually create routes per VLAN to make it go through the tunnel?

Some pieces of information that might be relevant

Ping attempt from Linux host A to Linux host B sniffed on Linux host B (the reply never reaches Linux host A)

14:22:58.924614 IP 192.168.101.3 > 192.168.189.101: ICMP echo request, id 20267, seq 1, length 64
14:22:58.924640 IP 192.168.189.101 > 192.168.101.3: ICMP echo reply, id 20267, seq 1, length 64

Ping attempt from pfSense to Linux host B sniffed on Linux host B (works fine)

14:23:20.770721 IP 10.42.42.2 > 192.168.189.101: ICMP echo request, id 58389, seq 0, length 64
14:23:20.770749 IP 192.168.189.101 > 10.42.42.2: ICMP echo reply, id 58389, seq 0, length 64

pfSense OpenVPN config

dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-256-CBC
auth SHA1
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.178.28
engine cryptodev
lport 51195
management /var/etc/openvpn/client1.sock unix
remote 192.168.178.13 51195
ifconfig 10.42.42.2 10.42.42.1
route 192.168.181.0 255.255.255.0
route 192.168.189.0 255.255.255.0
route 192.168.190.0 255.255.255.0
secret /var/etc/openvpn/client1.secret 
resolv-retry infinite

Route check on pfSense

route get 192.168.189.101
   route to: 192.168.189.101
destination: 192.168.189.0
       mask: 255.255.255.0
    gateway: 10.42.42.1
        fib: 0
  interface: ovpnc1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Route check on Linux host A (where 192.168.101.1 is the pfSense)

ip r get 192.168.189.101
192.168.189.101 via 192.168.101.1 dev enp3s4f0 src 192.168.101.3 
    cache

pfSense interfaces

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:a2:0d:62:ff
	inet6 fe80::208:a2ff:fe0d:62ff%lagg0 prefixlen 64 scopeid 0x13 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: lagg 
	laggproto loadbalance lagghash l2,l3,l4
	laggport: ix2 flags=4<ACTIVE>
	laggport: ix3 flags=4<ACTIVE>
lagg0.101: flags=88943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:08:a2:0d:62:ff
	inet6 fe80::208:a2ff:fe0d:62ff%lagg0.101 prefixlen 64 scopeid 0x17 
	inet 192.168.101.1 netmask 0xffffff00 broadcast 192.168.101.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	vlan: 101 vlanpcp: 0 parent interface: lagg0
	groups: vlan 
ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet6 fe80::208:a2ff:fe0d:62fd%ovpnc1 prefixlen 64 scopeid 0x18 
	inet 10.42.42.2 --> 10.42.42.1  netmask 0xffffffff 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun openvpn 
	Opened by PID 77923

viragomann

@tumble said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:

I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN?

Why do you think, it isn't?
Try a packet capture on the pfSense or the Edge routers VPN interface to be shure.

I guess, the routing problem is on the Edge. So check the routes there.

tumble

@viragomann said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:

I guess, the routing problem is on the Edge. So check the routes there.

Thanks for your reply, I think you are right. Debugging all possible things on different hops for hours got me confused and especially the different source IPs depending on whether the packet originates from the pfSense itself or from a VLAN behind it caught more of my attention than it should, I guess.

The packets flow through the ovpnc1 interface, from the pfSense itself and from the VLAN as well, the only difference being the source IP but I assume that's normal.

My EdgeRouter routing table looks like

root@edgerouter:~# ip r
0.0.0.0/24 dev vtun1  proto kernel  scope link 
default via 192.168.178.1 dev eth1  proto zebra 
10.0.142.1 dev vtun2  proto kernel  scope link 
10.0.142.2 dev vtun2  proto kernel  scope link  src 10.0.142.1 
10.6.66.0/24 dev vtun3  proto kernel  scope link  src 10.6.66.1 
10.7.0.1 dev vtun0  proto kernel  scope link  src 10.7.0.2 
10.7.0.2 dev vtun0  proto kernel  scope link 
10.9.0.0/24 dev vtun0  proto zebra 
10.10.0.0/24 dev vtun1  proto kernel  scope link  src 10.10.0.1 
10.15.0.0/24 dev vtun0  proto zebra 
10.42.42.1 dev vtun4  proto kernel  scope link 
10.42.42.2 dev vtun4  proto kernel  scope link  src 10.42.42.1 
192.168.10.0/24 dev eth2  proto kernel  scope link  src 192.168.10.10 
192.168.101.0/24 dev vtun4  proto zebra 
192.168.142.0/24 dev vtun2  proto zebra 
192.168.143.0/24 dev vtun2  proto zebra 
192.168.178.0/24 dev eth1  proto kernel  scope link  src 192.168.178.13 
192.168.180.0/24 dev vtun0  proto zebra 
192.168.181.0/24 dev eth3  proto kernel  scope link  src 192.168.181.1 
192.168.188.0/24 dev vtun0  proto zebra 
192.168.189.0/24 dev eth4  proto kernel  scope link  src 192.168.189.1 
192.168.190.0/24 dev eth6  proto kernel  scope link  src 192.168.190.1 
192.168.191.0/24 dev vtun0  proto zebra 
192.168.199.0/24 dev eth0  proto kernel  scope link  src 192.168.199.1 
192.168.200.0/24 dev eth5  proto kernel  scope link  src 192.168.200.1

and I would expect the response packet to 192.168.101.3 to go via vtun4 but according to this output

root@edgerouter:~# ip r get 192.168.101.3
192.168.101.3 via 192.168.178.28 dev eth1  src 192.168.189.1 
    cache

it doesn't. Takes the way via WAN iface. Didn't figure out yet how I can flush this cache to make sure it's not a caching issue, but I guess I will do some more research in that direction for now, unless someone here finds some conceptional mistake in my setup.

viragomann

I'm wondering why there is no gateway mentioned in the route for 192.168.101.0/24. 10.42.42.2 should by the gateway to route the traffic to.

Is the OpenVPN configured as a site-to-site?

tumble

I'm struggling quite a bit with understanding the EdgeRouter, which is one of the reasons I'm migrating everything to pfSense which feels much more intuitive to me.

There are two ways to set a route on the EdgeRouter - I can create a static route via UI (or CLI) which results in the entries you can see above and that basically works. I know that, because I have 2 more OpenVPN site-2-site tunnels, one to another EdgeRouter in a second office and one to a parallel in-house infrastructure behind another pfSense. The only difference between my 2 pfSenses, as far as I can tell, is that the new one uses VLANs (and has this fancy and a little confusing new switch, but I guess that's not involved here). Guess that's also what made me post in this section, because I'm relatively new to using them and thought I'm missing something here.
The other way of setting a route is by passing a parameter in the OpenVPN config (which is pretty much what pfSense seems to be doing). The route looks different in the table then, it's proto kernel instead of zebra. I tried that once while debugging but didn't have the feeling it solved anything. I might go that way once more and check if the route get shows a different result.

I'll show you the OpenVPN config on EdgeRouter side anyway, maybe you see something weird

 description "pfsense-redacted uplink"
 encryption aes256
 firewall {
     in {
         name PFSENSE-REDACTED-VPN
     }
     local {
         name PFSENSE-REDACTED-VPN
     }
     out {
         name PFSENSE-REDACTED-VPN
     }
 }
 local-address 10.42.42.1 {
 }
 local-port 51195
 mode site-to-site
 openvpn-option --float
 openvpn-option "--ping 10"
 openvpn-option "--ping-restart 20"
 openvpn-option --ping-timer-rem
 openvpn-option --persist-tun
 openvpn-option --persist-key
 openvpn-option "--user nobody"
 openvpn-option "--group nogroup"
 openvpn-option ""
 openvpn-option "--verb 4"
 remote-address 10.42.42.2
 remote-host 192.168.178.28
 remote-port 51195
 shared-secret-key-file /config/auth/pfsense-redacted/secret

Derelict

That's all great but this is not edgerouter support.

It appears the pfSense side is fine but the edgerouter is not routing traffic for 192.168.101.0/24 back over the tunnel.

That said, try adding an OpenVPN option on the edgerouter that results in this:

"--route 192.168.101.0 255.255.255.0"

edit -

Probably not since the zebra route is in the table to the correct tunnel it must be getting that from somewhere else. Probably have to ask them.

OpenVPN site-2-site - VLAN host cannot receive response from the other end

What works

What doesn't work

What I already figured out

Some pieces of information that might be relevant

Products

Services

Support

News

Resources

Company

Our Mission

Subscribe to our Newsletter