OpenVPN client, routes being ignored
-
Do you have policy-routing rules on LAN that direct traffic to a particular gateway?
If so, then you need some ordinary pass rules above those, that match the traffic to those internal subnets that are reached across the site-2-site VPN. That will let the traffic be handed to the routing table.Not that I know of - I have dual wan with failover, and I removed all of that to rule out any odd issues there. I went through all the rules tabs to see if any rules defined a gateway and none do. I have tried auto and manual outbound NAT as well, even tried a manual NAT rule that specifically works on a src of the LAN net and destination of my remote net and uses the ovpnc3 IP as the NAT interface.
What really puzzles me here is I can't figure out where the traffic is going. If I do a packet capture in the web GUI or with tcpdump in the shell, I see my pings ONLY on ovpnc3. If I do a tcpdump on the other end of the vpn (on the tun interface) I don't see the traffic. If I then start pinging from the client's shell, I do see that traffic on the server's tun interface.
[2.1.4-RELEASE][admin@gw.xx.com]/root(87): tcpdump -ni ovpnc3 src or dst 10.88.77.37 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ovpnc3, link-type NULL (BSD loopback), capture size 96 bytes 13:45:43.859136 IP 10.3.2.41 > 10.88.77.37: ICMP echo request, id 39442, seq 215, length 64 13:45:44.860266 IP 10.3.2.41 > 10.88.77.37: ICMP echo request, id 39442, seq 216, length 64 13:45:45.860712 IP 10.3.2.41 > 10.88.77.37: ICMP echo request, id 39442, seq 217, length 64 13:45:46.859402 IP 10.3.2.41 > 10.88.77.37: ICMP echo request, id 39442, seq 218, length 64And nothing showing here:
[root@trunk /usr/local/etc/openvpn]# tcpdump -ni tun0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type NULL (BSD loopback), capture size 96 bytesAnd then I start a ping to the same IP from the pfsense shell:
[2.1.4-RELEASE][admin@gw.xx.com]/root(88): ping 10.88.77.37 PING 10.88.77.37 (10.88.77.37): 56 data bytes 64 bytes from 10.88.77.37: icmp_seq=0 ttl=64 time=11.605 ms 64 bytes from 10.88.77.37: icmp_seq=1 ttl=64 time=12.337 ms 64 bytes from 10.88.77.37: icmp_seq=2 ttl=64 time=11.733 ms ^C --- 10.88.77.37 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 11.605/11.892/12.337/0.319 ms [2.1.4-RELEASE][admin@gw.xx.com]/root(89):And I see this in that same tcpdump session:
[root@trunk /usr/local/etc/openvpn]# tcpdump -ni tun0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type NULL (BSD loopback), capture size 96 bytes 13:48:37.641992 IP 10.99.0.101 > 10.88.77.37: ICMP echo request, id 43782, seq 0, length 64 13:48:37.642014 IP 10.88.77.37 > 10.99.0.101: ICMP echo reply, id 43782, seq 0, length 64 13:48:38.641176 IP 10.99.0.101 > 10.88.77.37: ICMP echo request, id 43782, seq 1, length 64 13:48:38.641196 IP 10.88.77.37 > 10.99.0.101: ICMP echo reply, id 43782, seq 1, length 64 13:48:39.643422 IP 10.99.0.101 > 10.88.77.37: ICMP echo request, id 43782, seq 2, length 64 13:48:39.643435 IP 10.88.77.37 > 10.99.0.101: ICMP echo reply, id 43782, seq 2, length 64Totally maddening.
I do feel like the source address difference could be doing something, but I'm not sure what (ie: 10.3.2.41 on my LAN vs. 10.99.0.101 when pinging from the shell). If I saw one-way traffic, then I'd want to start digging more into NAT or verifiying the return route is OK, but not seeing the traffic from the LAN follow the route to ovpnc3 just puzzles me.
And another client connection on the same pfsense box works without issue. Server side there is the same, just a FreeBSD box with OpenVPN.
Also, if anything the memory/resource use is better in 2.2.* than in 2.1.* pfSense releases. Actually I would be inclined to upgrade on old hardware. Just make sure to have a backup just in case some really ancient bit of hardware no longer works.
I hear you, I just have not gone beyond FreeBSD 9.3 anywhere, and I know that each release tends to be tested less and less on older hardware (which is fine - how many P-III hosts are still in service anywhere). If all else fails, I'll try that by just sticking a clean drive in this host and doing a 2.2.mumble install on there.
-
Someone else with a good idea please post! I can't think what can be wrong here. You are seeing packets leave ovpnc3 but a dump running on the server end never sees them arrive. That should happen before any firewall rules at the server end. And how can those ICMP packets be selective "lost" across the OpenVPN circuit - they are the same length as other ICMP that get through.
Use traceroute to try and see the path that packets are taking - but from the tcpdump evidence they are leaving the correct OpenVPN interface.
-
Did you try with something else than ping?
-
Yep, same deal.
-
Yep, same deal.
You have something to examine the packets INSIDE the vpn…...........NSA maybe? If your sniffing the vpn you won't see anything, think encrypted (not without some really, really expensive hardware software and more smarts than I have)....packets go in......to somewhere..................Check at somewhere end maybe?
-
Upgraded to 2.2.1, same deal.
Very odd, here's more proof that traffic from the LAN is just not being sent over the openvpn interface:
fruitcake:resourceFurniture spork$ ping 10.88.77.37 PING 10.88.77.37 (10.88.77.37): 56 data bytes 36 bytes from 10.240.176.233: Communication prohibited by filter Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 035a 0 0000 40 01 13a7 10.3.2.41 10.88.77.37 Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3The error there is from 10.240.176.233, which is a router a few hops out from my cable modem, so clearly this traffic is headed out the WAN connection. I still see the proper routes in netstat output and can ping the same IP from the pfsense console.
-
Given the description, you have to be forcing that traffic to the Internet with your policy routing rules.
-
I was seeing the problem with and without the only policy routing I know of (failover gateway) in place. The ICMP error message, that you are correct about. When I removed the failover rule, that behavior stopped, but I still was back where I started - I can see traffic go out the ovpnc3 tun interface, but never see it on the other end.
I'm on to something though. On the server, I cranked the verbosity all the way up on openvpn and saw this:
Apr 3 17:40:47 trunk openvpn[19784]: spork/x.x.x.x:41816 MULTI: bad source address from client [10.3.2.41], packet droppedWith either auto outbound NAT rules enabled or "hybrid" rules that include a manual NAT from the LAN to the remote LAN using the openvpn TUN IP, or manual only NAT rules, I notice that nothing changes - the remote end still sees my LAN IP as the source, not the TUN IP, which is what I see when pinging from the console. Maybe it's just not an option to NAT on the way out - the OpenVPN client interfaces never show up as real "interfaces" in the various config screens where there's an interface selection.
Regarding policy routing, shouldn't creating an OpenVPN client instance auto-generate a LAN rule to deal with that?
-
I think you need to post your LAN rules to get more eyes on them.
-
I've come up with a big of a workaround that seems to work and makes everything a little easier to understand.
I've added a new interface and tied it to the OpenVPN client. In addtion to that, I've added the following rules:
- Outbound LAN, set gateway to this new openvpn interface
- Outbound NAT, set to manual add a rule that when src is LAN dst is my remote subnets, interface is new openvpn interface, NAT using the openvpn local IP
- On far end, add a pf rule to NAT on the openvpn interface's IPs
Very convoluted, but it works. I'm hoping that interface assignment "sticks" across reboots and creating/deleting other openvpn clients.
-
The "bad source address from client" means the routing on that instance of OpenVPN generating that log is wrong, it thinks that's not a valid source network for that client. The NAT mess will work around that problem because it hides the real source address, and it will persist regardless of adding/removing/editing OpenVPN instances as those interface identifiers are static, but you should really fix the routing issue and get rid of the NAT.
-
@cmb:
The "bad source address from client" means the routing on that instance of OpenVPN generating that log is wrong, it thinks that's not a valid source network for that client.
Yep, that one was an easy google, if i added an "iroute" statement to the config for this client, that fixes everything up.
@cmb:
The NAT mess will work around that problem because it hides the real source address, and it will persist regardless of adding/removing/editing OpenVPN instances as those interface identifiers are static, but you should really fix the routing issue and get rid of the NAT.
I'd rather not, to be honest. :) While I generally "trust" the remote network I'm going into, I don't trust a totally firewall-bypassed network to network link, so I actually prefer getting NAT involved. I also like only having to think about a single IP when getting traffic back over the tunnel as opposed to getting my home internal /24 into the remote network's routing mesh.
There's probably a better way to do this, but I can't come up with one. It got bad enough that I started thinking of moving to IPSEC, which makes me shudder.
-
NAT is not a security mechanism, you can accomplish exactly the same thing with firewall rules and no NAT.
