Policy Routing via OpenVPN Uplink
-
Hi,
I'm having a bit of problem with a policy routing via an OpenVPN uplink. I hope someone can help me. Here's the setup:
A SG-7100 is connected to the internet via a static subnet on Interface lagg0.4090
Additionally we're using a peer-to-peer OpenVPN tunnel to a router in a datacenter in order to route a couple of additional public IP subnets to our on-site firewall.
We've been doing this for a couple of years now with a linux based router/firewall, but we're now in the process of migrating that setup to our new SG-7100.
The incoming traffic is being routed via a openvpn transfer net (172.29.2.0/24) to our pfSense and is then being forwarded via another internal transfer net to a host that is using one of our public addresses from that datacenter subnet: 95.216.47.180
The incoming traffic reaches our host just fine, as we can see with tcpdump on the host:
root@core-hg:~# tcpdump -n -i ens160 |grep 95.216.47.180 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 12:10:02.882885 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33015, length 44 12:10:02.882904 ARP, Request who-has 192.168.100.250 tell 95.216.47.180, length 28 12:10:02.883350 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33015, length 44 12:10:02.985654 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33016, length 44 12:10:02.985665 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33016, length 44 12:10:03.086816 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33017, length 44 12:10:03.086828 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33017, length 44 12:10:03.187353 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33018, length 44In this example I'm pinging the host (95.216.47.180) from my home office (217.240.147.187)
As you can see the host generates reply packages and routes them back via the transfer net to our SG-7100. I can see the reply packages on the SG-7100:
[2.4.4-RELEASE][root@netgate1.wittich-hoehr.de]/root: tcpdump -n -i ix0 |grep 95.216.47.180 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:12:47.177766 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33015, length 44 12:12:47.177832 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33015, length 44 12:12:47.276528 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33016, length 44 12:12:47.276602 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33016, length 44 12:12:47.379494 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33017, length 44 12:12:47.379566 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33017, length 44So, up until this point everything works as expected.
But despite a policy rule that should route traffic from 95.216.47.160/27 back over the OpenVPN link, the SG-7100 is actually routing those reply packages via it's regular default nexthop on lagg0.4090:
2.4.4-RELEASE][root@netgate1.wittich-hoehr.de]/root: tcpdump -n -i lagg0.4090 | grep 95.216.47.180 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lagg0.4090, link-type EN10MB (Ethernet), capture size 262144 bytes 12:16:00.254283 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33015, length 44 12:16:00.356478 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33016, length 44 12:16:00.461685 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33017, length 44 12:16:00.564703 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33018, length 44 12:16:00.668986 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33019, length 44As a result the reply packages are being discarded by our local ISP and never reach my home office.
One could think that I just made a mistake setting up the policy route, but I'm pretty sure the rule is correct, because outgoing traffic from 95.216.47.180 that is being INITIATED here is being routed correctly via the OpenVPN uplink:
root@core-hg:~# traceroute -n -s 95.216.47.180 1.1.1.1 traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 172.29.2.1 32.577 ms 32.648 ms 32.640 ms 2 95.216.14.65 32.881 ms 32.879 ms 33.085 ms 3 213.239.224.133 32.828 ms 32.832 ms 32.808 ms 4 213.239.252.102 39.615 ms 39.714 ms 213.239.224.17 39.603 ms 5 194.68.128.246 38.930 ms 194.68.123.246 39.949 ms 40.030 ms 6 1.1.1.1 40.883 ms 40.647 ms 39.679 msAs you can see the traffic leaves through our OpenVPN transfer net and reaches it's destination. Also the reply packages from 1.1.1.1 are being routed correctly back through that OpenVPN uplink.
So, I would say, that the policy rule works in general, but I must have missed something.
Do you have any idea why traffic that is being INITIATED by our host is being routed correctly through OpenVPN while REPLY traffic is going through the regular default route?
-
In case someone is interested: I solved the problem by switching from a routed OpenVPN (point-to-point-routing) to a bridged layer2-OpenVPN link. By using a bridge interface with a corresponding uplink gateway it works like a charm.
Anyway: If some of the developers are reading this: it would be nice if someone would check if this is a bug with routed OpenVPN links. In theory my initial approach of using a policy based rule to route the traffic over the OpenVPN link should work. But it doesn't. Even after taking the additional transfer net to the internal router out of the equation it didn't work after I assigned the public IP to one of Netgate's own interfaces. Outgoing traffic from that interface was routed correctly via the OpenVPN link, but reply traffic from incoming connections always got routed via the default next hop instead of the OpenVPN tunnel. I spent almost 8 hours troubleshooting but didn't find a solution other than switching to a bridged OpenVPN link.
On a site note: I'm using a failover setup with CARP IP addresses in case that makes any difference.
If you need help in reproducing this presumed bug, don't hesitate to contact me.
Regards,
Jörn Bredereck -
@jbredereck said in Policy Routing via OpenVPN Uplink:
Do you have any idea why traffic that is being INITIATED by our host is being routed correctly through OpenVPN while REPLY traffic is going through the regular default route?
Hi
This happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti,gre ...)
The original traffic that came through the virtual tunnel will never return through the same gateway back.
The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnel
For example , the request
8.8.8.8->WAN2_IP(8.8.8.8 port forwarding to YOUR INTERNAL HOST)>10.10.10.1 (outbound NAT 8.8.8.8 -> 10.10.10.1) ->10.10.10.2(10.10.10.1)->LAN INTERFACE(10.10.10.1)->YOUR INTERNAL HOST(10.10.10.1)
The reply
YOUR INTERNAL HOST(10.10.10.1)->LAN INTERFACE(10.10.10.1)->10.10.10.2(10.10.10.1)->10.10.10.1(10.10.10.1->8.8.8.8)->>WAN2_IP(8.8.8.8)->8.8.8.8This is an example of how I have configured such a scheme in Linux+PFSense (Linux side of the tunnel)
Prerouting - port forwarding
Postrouting - NAT outbound
192.168.1.230 - internal host
10.10.100.2 - tunnel ip address:PREROUTING ACCEPT [0:0] -A PREROUTING -d XXX.XXX.XXX.XXX -p tcp -m multiport --destination-port 25,465,587,993 -j DNAT --to-destination 192.168.1.230 :POSTROUTING ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A POSTROUTING -o tun100 -p tcp -m multiport --destination-port 25,465,587,993 -d 192.168.1.230 -j SNAT --to-source 10.10.100.2 -
@Konstanti said in Policy Routing via OpenVPN Uplink:
This happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti ...)
The traffic that came through the virtual tunnel will never return through the same gateway back.
The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnelThanks for your answer. So we're talking about a known bug here which might get fixed in the future, or are we talking about a general limitation of how virtual interfaces under FreeBSD/pfSense work? Because policy based routing with tun-interface in Linux is something we use at dozens of our clients, and we were planing to migrate to pfSense/netgate some of those installations in the future.
As for the NAT-workaround: Yes, that works in theory; i tried that out on the weekend already. But unfortunately that's no solution, since we're hosting web- and mail servers behind that pfSense firewall, and all those servers need the original public source IP address for things like logging and authorization. Simply NATing all source IPs to our own routers interface would result in having this IP in all our web servers access logs for example, and things like blacklisting certain SMTP source addresses in our MX wouldn't work any more.
For now I will go with the bridged tap interfaces for the OpenVPN links. That works for now, but I would certainly prefer layer3-routed OpenVPN links instead of layer2-bridging accross the internet.
-
@jbredereck
Unfortunately , I don't know what will happen in the future )).
This problem has long been known . In my opinion, since OpenBsd and still not resolved . -
@Konstanti said in Policy Routing via OpenVPN Uplink:
@jbredereck
Unfortunately , I don't know what will happen in the future )).
This problem has long been known . In my opinion, since OpenBsd and still not resolved .So this is a bug/limitation in FreeBSD and nothing that the pfSense-developers could fix, right?
That's a shame because other than that I'm really thrilled about pfSense and the netgate boxes. Unfortunately we're using OpenVPN quiet a lot.
Would another tunnel protocol be a solution to this problem? Or does the same limitation apply to GRE and IPSec tunnels?
-
@jbredereck said in Policy Routing via OpenVPN Uplink:
So this is a bug/limitation in FreeBSD and nothing that the pfSense-developers could fix, right?
That's a shame because other than that I'm really thrilled about pfSense and the netgate boxes. Unfortunately we're using OpenVPN quiet a lot.
Would another tunnel protocol be a solution to this problem? Or does the same limitation apply to GRE and IPSec tunnels?In my opinion, this restriction applies to any virtual routed connection
In non-routable connections where used, the traffic selectors (ipec+ikev2 or ipsec+ikev1) these restrictions should not be. But in your case it will be very difficult to configure and difficult to manage all these. -
Alright, thanks for all the info. I will stick to bridged OpenVPN links for now.
-
@jbredereck I got same problem. I was trying to use IPsec tunnel with routed(VTI) mode. I stuck with same situation that machine behind vpn get incoming packet, but reply was going out through wrong interface (default instead of tunnel).
Your solution saved my time.
Thanks!Regards,
Jacek