Policy Routing via OpenVPN Uplink



  • Hi,

    I'm having a bit of problem with a policy routing via an OpenVPN uplink. I hope someone can help me. Here's the setup:

    A SG-7100 is connected to the internet via a static subnet on Interface lagg0.4090

    Additionally we're using a peer-to-peer OpenVPN tunnel to a router in a datacenter in order to route a couple of additional public IP subnets to our on-site firewall.

    We've been doing this for a couple of years now with a linux based router/firewall, but we're now in the process of migrating that setup to our new SG-7100.

    The incoming traffic is being routed via a openvpn transfer net (172.29.2.0/24) to our pfSense and is then being forwarded via another internal transfer net to a host that is using one of our public addresses from that datacenter subnet: 95.216.47.180

    The incoming traffic reaches our host just fine, as we can see with tcpdump on the host:

    root@core-hg:~# tcpdump -n -i ens160 |grep 95.216.47.180
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
    12:10:02.882885 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33015, length 44
    12:10:02.882904 ARP, Request who-has 192.168.100.250 tell 95.216.47.180, length 28
    12:10:02.883350 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33015, length 44
    12:10:02.985654 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33016, length 44
    12:10:02.985665 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33016, length 44
    12:10:03.086816 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33017, length 44
    12:10:03.086828 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35634, seq 33017, length 44
    12:10:03.187353 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35634, seq 33018, length 44
    

    In this example I'm pinging the host (95.216.47.180) from my home office (217.240.147.187)

    As you can see the host generates reply packages and routes them back via the transfer net to our SG-7100. I can see the reply packages on the SG-7100:

    [2.4.4-RELEASE][root@netgate1.wittich-hoehr.de]/root: tcpdump -n -i ix0 |grep 95.216.47.180
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on ix0, link-type EN10MB (Ethernet), capture size 262144 bytes
    12:12:47.177766 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33015, length 44
    12:12:47.177832 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33015, length 44
    12:12:47.276528 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33016, length 44
    12:12:47.276602 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33016, length 44
    12:12:47.379494 IP 217.240.147.187 > 95.216.47.180: ICMP echo request, id 35651, seq 33017, length 44
    12:12:47.379566 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35651, seq 33017, length 44
    

    So, up until this point everything works as expected.

    But despite a policy rule that should route traffic from 95.216.47.160/27 back over the OpenVPN link, the SG-7100 is actually routing those reply packages via it's regular default nexthop on lagg0.4090:

    2.4.4-RELEASE][root@netgate1.wittich-hoehr.de]/root: tcpdump -n -i lagg0.4090 | grep 95.216.47.180
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on lagg0.4090, link-type EN10MB (Ethernet), capture size 262144 bytes
    12:16:00.254283 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33015, length 44
    12:16:00.356478 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33016, length 44
    12:16:00.461685 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33017, length 44
    12:16:00.564703 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33018, length 44
    12:16:00.668986 IP 95.216.47.180 > 217.240.147.187: ICMP echo reply, id 35671, seq 33019, length 44
    

    As a result the reply packages are being discarded by our local ISP and never reach my home office.

    One could think that I just made a mistake setting up the policy route, but I'm pretty sure the rule is correct, because outgoing traffic from 95.216.47.180 that is being INITIATED here is being routed correctly via the OpenVPN uplink:

    root@core-hg:~# traceroute -n -s 95.216.47.180 1.1.1.1
    traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
     1  172.29.2.1  32.577 ms  32.648 ms  32.640 ms
     2  95.216.14.65  32.881 ms  32.879 ms  33.085 ms
     3  213.239.224.133  32.828 ms  32.832 ms  32.808 ms
     4  213.239.252.102  39.615 ms  39.714 ms 213.239.224.17  39.603 ms
     5  194.68.128.246  38.930 ms 194.68.123.246  39.949 ms  40.030 ms
     6  1.1.1.1  40.883 ms  40.647 ms  39.679 ms
    

    As you can see the traffic leaves through our OpenVPN transfer net and reaches it's destination. Also the reply packages from 1.1.1.1 are being routed correctly back through that OpenVPN uplink.

    So, I would say, that the policy rule works in general, but I must have missed something.

    Do you have any idea why traffic that is being INITIATED by our host is being routed correctly through OpenVPN while REPLY traffic is going through the regular default route?



  • In case someone is interested: I solved the problem by switching from a routed OpenVPN (point-to-point-routing) to a bridged layer2-OpenVPN link. By using a bridge interface with a corresponding uplink gateway it works like a charm.

    Anyway: If some of the developers are reading this: it would be nice if someone would check if this is a bug with routed OpenVPN links. In theory my initial approach of using a policy based rule to route the traffic over the OpenVPN link should work. But it doesn't. Even after taking the additional transfer net to the internal router out of the equation it didn't work after I assigned the public IP to one of Netgate's own interfaces. Outgoing traffic from that interface was routed correctly via the OpenVPN link, but reply traffic from incoming connections always got routed via the default next hop instead of the OpenVPN tunnel. I spent almost 8 hours troubleshooting but didn't find a solution other than switching to a bridged OpenVPN link.

    On a site note: I'm using a failover setup with CARP IP addresses in case that makes any difference.

    If you need help in reproducing this presumed bug, don't hesitate to contact me.

    Regards,
    Jörn Bredereck



  • @jbredereck said in Policy Routing via OpenVPN Uplink:

    Do you have any idea why traffic that is being INITIATED by our host is being routed correctly through OpenVPN while REPLY traffic is going through the regular default route?

    Hi
    This happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti,gre ...)
    The original traffic that came through the virtual tunnel will never return through the same gateway back.
    The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnel
    For example , the request
    8.8.8.8->WAN2_IP(8.8.8.8 port forwarding to YOUR INTERNAL HOST)>10.10.10.1 (outbound NAT 8.8.8.8 -> 10.10.10.1) ->10.10.10.2(10.10.10.1)->LAN INTERFACE(10.10.10.1)->YOUR INTERNAL HOST(10.10.10.1)
    The reply
    YOUR INTERNAL HOST(10.10.10.1)->LAN INTERFACE(10.10.10.1)->10.10.10.2(10.10.10.1)->10.10.10.1(10.10.10.1->8.8.8.8)->>WAN2_IP(8.8.8.8)->8.8.8.8

    This is an example of how I have configured such a scheme in Linux+PFSense (Linux side of the tunnel)
    Prerouting - port forwarding
    Postrouting - NAT outbound
    192.168.1.230 - internal host
    10.10.100.2 - tunnel ip address

    :PREROUTING ACCEPT [0:0]
    -A PREROUTING -d XXX.XXX.XXX.XXX -p tcp -m multiport --destination-port 25,465,587,993 -j DNAT --to-destination 192.168.1.230
    :POSTROUTING ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A POSTROUTING -o tun100 -p tcp -m multiport --destination-port 25,465,587,993 -d 192.168.1.230 -j SNAT --to-source 10.10.100.2
    


  • @Konstanti said in Policy Routing via OpenVPN Uplink:

    This happens because pfsense is also not working option "reply-to" for virtual interfaces (e.g., openvpn,vti ...)
    The traffic that came through the virtual tunnel will never return through the same gateway back.
    The solution to this problem is to use outbound NAT for this traffic on the other end of the tunnel

    Thanks for your answer. So we're talking about a known bug here which might get fixed in the future, or are we talking about a general limitation of how virtual interfaces under FreeBSD/pfSense work? Because policy based routing with tun-interface in Linux is something we use at dozens of our clients, and we were planing to migrate to pfSense/netgate some of those installations in the future.

    As for the NAT-workaround: Yes, that works in theory; i tried that out on the weekend already. But unfortunately that's no solution, since we're hosting web- and mail servers behind that pfSense firewall, and all those servers need the original public source IP address for things like logging and authorization. Simply NATing all source IPs to our own routers interface would result in having this IP in all our web servers access logs for example, and things like blacklisting certain SMTP source addresses in our MX wouldn't work any more.

    For now I will go with the bridged tap interfaces for the OpenVPN links. That works for now, but I would certainly prefer layer3-routed OpenVPN links instead of layer2-bridging accross the internet.



  • @jbredereck
    Unfortunately , I don't know what will happen in the future )).
    This problem has long been known . In my opinion, since OpenBsd and still not resolved .



  • @Konstanti said in Policy Routing via OpenVPN Uplink:

    @jbredereck
    Unfortunately , I don't know what will happen in the future )).
    This problem has long been known . In my opinion, since OpenBsd and still not resolved .

    So this is a bug/limitation in FreeBSD and nothing that the pfSense-developers could fix, right?

    That's a shame because other than that I'm really thrilled about pfSense and the netgate boxes. Unfortunately we're using OpenVPN quiet a lot.

    Would another tunnel protocol be a solution to this problem? Or does the same limitation apply to GRE and IPSec tunnels?



  • @jbredereck said in Policy Routing via OpenVPN Uplink:

    So this is a bug/limitation in FreeBSD and nothing that the pfSense-developers could fix, right?
    That's a shame because other than that I'm really thrilled about pfSense and the netgate boxes. Unfortunately we're using OpenVPN quiet a lot.
    Would another tunnel protocol be a solution to this problem? Or does the same limitation apply to GRE and IPSec tunnels?

    In my opinion, this restriction applies to any virtual routed connection
    In non-routable connections where used, the traffic selectors (ipec+ikev2 or ipsec+ikev1) these restrictions should not be. But in your case it will be very difficult to configure and difficult to manage all these.



  • @Konstanti

    Alright, thanks for all the info. I will stick to bridged OpenVPN links for now.



  • @jbredereck I got same problem. I was trying to use IPsec tunnel with routed(VTI) mode. I stuck with same situation that machine behind vpn get incoming packet, but reply was going out through wrong interface (default instead of tunnel).

    Your solution saved my time.
    Thanks!

    Regards,
    Jacek


Log in to reply