OpenVPN site-2-site - VLAN host cannot receive response from the other end
-
Hi,
I chose this section although I must admit I'm not sure it's a VLAN-related issue.
I'm currently setting up a new network in parallel to an old one and to have a connection between these two during migration phase, I established an OpenVPN site-2-site tunnel.
I really hope I'm not making a fool of myself because I missed something obvious here. Anyway, I'd be happy to get this working. :)
This figure illustrates the setup:
What works
- the pfSense from network A can ping Linux host B in network B via site-2-site tunnel
- from the pfSense I can initiate a SSH connection to Linux host B
What doesn't work
- Linux host A from network A cannot reach Linux host B in network B or to be more precise, the ICMP packet gets through to Linux host B but the reply doesn't find the way back
What I already figured out
- connections from the pfSense to Linux host B have source IP 10.42.42.2 (the OpenVPN endpoint)
- connections from Linux host A to Linux host B have source IP 192.168.101.3
- the result is that response packets back to the pfSense go through the tunnel while response attempts to Linux host A go through the non-tunnel network (and that's not working and expected)
Now you might say it's a routing issue, but what I don't understand is, why does the pfSense behave differently depending on the source of the packets? I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN? Do I need to manually create routes per VLAN to make it go through the tunnel?
Some pieces of information that might be relevant
Ping attempt from Linux host A to Linux host B sniffed on Linux host B (the reply never reaches Linux host A)
14:22:58.924614 IP 192.168.101.3 > 192.168.189.101: ICMP echo request, id 20267, seq 1, length 64 14:22:58.924640 IP 192.168.189.101 > 192.168.101.3: ICMP echo reply, id 20267, seq 1, length 64
Ping attempt from pfSense to Linux host B sniffed on Linux host B (works fine)
14:23:20.770721 IP 10.42.42.2 > 192.168.189.101: ICMP echo request, id 58389, seq 0, length 64 14:23:20.770749 IP 192.168.189.101 > 10.42.42.2: ICMP echo reply, id 58389, seq 0, length 64
pfSense OpenVPN config
dev ovpnc1 verb 1 dev-type tun dev-node /dev/tun1 writepid /var/run/openvpn_client1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA1 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 192.168.178.28 engine cryptodev lport 51195 management /var/etc/openvpn/client1.sock unix remote 192.168.178.13 51195 ifconfig 10.42.42.2 10.42.42.1 route 192.168.181.0 255.255.255.0 route 192.168.189.0 255.255.255.0 route 192.168.190.0 255.255.255.0 secret /var/etc/openvpn/client1.secret resolv-retry infinite
Route check on pfSense
route get 192.168.189.101 route to: 192.168.189.101 destination: 192.168.189.0 mask: 255.255.255.0 gateway: 10.42.42.1 fib: 0 interface: ovpnc1 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
Route check on Linux host A (where 192.168.101.1 is the pfSense)
ip r get 192.168.189.101 192.168.189.101 via 192.168.101.1 dev enp3s4f0 src 192.168.101.3 cache
pfSense interfaces
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:08:a2:0d:62:ff inet6 fe80::208:a2ff:fe0d:62ff%lagg0 prefixlen 64 scopeid 0x13 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto loadbalance lagghash l2,l3,l4 laggport: ix2 flags=4<ACTIVE> laggport: ix3 flags=4<ACTIVE> lagg0.101: flags=88943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:08:a2:0d:62:ff inet6 fe80::208:a2ff:fe0d:62ff%lagg0.101 prefixlen 64 scopeid 0x17 inet 192.168.101.1 netmask 0xffffff00 broadcast 192.168.101.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect status: active vlan: 101 vlanpcp: 0 parent interface: lagg0 groups: vlan ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 options=80000<LINKSTATE> inet6 fe80::208:a2ff:fe0d:62fd%ovpnc1 prefixlen 64 scopeid 0x18 inet 10.42.42.2 --> 10.42.42.1 netmask 0xffffffff nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: tun openvpn Opened by PID 77923
-
@tumble said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:
I mean, why isn't traffic to the remote net (which I configured on the pfSense side in the OpenVPN config) routed through the VPN?
Why do you think, it isn't?
Try a packet capture on the pfSense or the Edge routers VPN interface to be shure.I guess, the routing problem is on the Edge. So check the routes there.
-
@viragomann said in OpenVPN site-2-site - VLAN host cannot receive response from the other end:
I guess, the routing problem is on the Edge. So check the routes there.
Thanks for your reply, I think you are right. Debugging all possible things on different hops for hours got me confused and especially the different source IPs depending on whether the packet originates from the pfSense itself or from a VLAN behind it caught more of my attention than it should, I guess.
The packets flow through the ovpnc1 interface, from the pfSense itself and from the VLAN as well, the only difference being the source IP but I assume that's normal.
My EdgeRouter routing table looks like
root@edgerouter:~# ip r 0.0.0.0/24 dev vtun1 proto kernel scope link default via 192.168.178.1 dev eth1 proto zebra 10.0.142.1 dev vtun2 proto kernel scope link 10.0.142.2 dev vtun2 proto kernel scope link src 10.0.142.1 10.6.66.0/24 dev vtun3 proto kernel scope link src 10.6.66.1 10.7.0.1 dev vtun0 proto kernel scope link src 10.7.0.2 10.7.0.2 dev vtun0 proto kernel scope link 10.9.0.0/24 dev vtun0 proto zebra 10.10.0.0/24 dev vtun1 proto kernel scope link src 10.10.0.1 10.15.0.0/24 dev vtun0 proto zebra 10.42.42.1 dev vtun4 proto kernel scope link 10.42.42.2 dev vtun4 proto kernel scope link src 10.42.42.1 192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.10 192.168.101.0/24 dev vtun4 proto zebra 192.168.142.0/24 dev vtun2 proto zebra 192.168.143.0/24 dev vtun2 proto zebra 192.168.178.0/24 dev eth1 proto kernel scope link src 192.168.178.13 192.168.180.0/24 dev vtun0 proto zebra 192.168.181.0/24 dev eth3 proto kernel scope link src 192.168.181.1 192.168.188.0/24 dev vtun0 proto zebra 192.168.189.0/24 dev eth4 proto kernel scope link src 192.168.189.1 192.168.190.0/24 dev eth6 proto kernel scope link src 192.168.190.1 192.168.191.0/24 dev vtun0 proto zebra 192.168.199.0/24 dev eth0 proto kernel scope link src 192.168.199.1 192.168.200.0/24 dev eth5 proto kernel scope link src 192.168.200.1
and I would expect the response packet to 192.168.101.3 to go via vtun4 but according to this output
root@edgerouter:~# ip r get 192.168.101.3 192.168.101.3 via 192.168.178.28 dev eth1 src 192.168.189.1 cache
it doesn't. Takes the way via WAN iface. Didn't figure out yet how I can flush this cache to make sure it's not a caching issue, but I guess I will do some more research in that direction for now, unless someone here finds some conceptional mistake in my setup.
-
I'm wondering why there is no gateway mentioned in the route for 192.168.101.0/24. 10.42.42.2 should by the gateway to route the traffic to.
Is the OpenVPN configured as a site-to-site?
-
I'm struggling quite a bit with understanding the EdgeRouter, which is one of the reasons I'm migrating everything to pfSense which feels much more intuitive to me.
There are two ways to set a route on the EdgeRouter - I can create a static route via UI (or CLI) which results in the entries you can see above and that basically works. I know that, because I have 2 more OpenVPN site-2-site tunnels, one to another EdgeRouter in a second office and one to a parallel in-house infrastructure behind another pfSense. The only difference between my 2 pfSenses, as far as I can tell, is that the new one uses VLANs (and has this fancy and a little confusing new switch, but I guess that's not involved here). Guess that's also what made me post in this section, because I'm relatively new to using them and thought I'm missing something here.
The other way of setting a route is by passing a parameter in the OpenVPN config (which is pretty much what pfSense seems to be doing). The route looks different in the table then, it's proto kernel instead of zebra. I tried that once while debugging but didn't have the feeling it solved anything. I might go that way once more and check if the route get shows a different result.I'll show you the OpenVPN config on EdgeRouter side anyway, maybe you see something weird
description "pfsense-redacted uplink" encryption aes256 firewall { in { name PFSENSE-REDACTED-VPN } local { name PFSENSE-REDACTED-VPN } out { name PFSENSE-REDACTED-VPN } } local-address 10.42.42.1 { } local-port 51195 mode site-to-site openvpn-option --float openvpn-option "--ping 10" openvpn-option "--ping-restart 20" openvpn-option --ping-timer-rem openvpn-option --persist-tun openvpn-option --persist-key openvpn-option "--user nobody" openvpn-option "--group nogroup" openvpn-option "" openvpn-option "--verb 4" remote-address 10.42.42.2 remote-host 192.168.178.28 remote-port 51195 shared-secret-key-file /config/auth/pfsense-redacted/secret
-
That's all great but this is not edgerouter support.
It appears the pfSense side is fine but the edgerouter is not routing traffic for 192.168.101.0/24 back over the tunnel.
That said, try adding an OpenVPN option on the edgerouter that results in this:
"--route 192.168.101.0 255.255.255.0"
edit -
Probably not since the zebra route is in the table to the correct tunnel it must be getting that from somewhere else. Probably have to ask them.