OpenVPN performance - Huge Improvement
-
Hi all,
I'm fighting with bad openVPN performance for a while now. See this post over at the openVPN forum: https://forums.openvpn.net/post21446.html
During my testing, I noticed a huge performance jump when setting net.inet.ip.fastforwarding = 1, followed by a reboot.
This setting can be found under System|Advanced|System Tunables and defaults to 0.
-
I've done some iperf testing to validate my assertions that the setting vastly improves openVPN Server performance on pfSense.
Network and Test Layout:
IPerf: [Client]-----------------------------------------------------------------------------------[Server] VPN: [OpenVPN Client]---------------------------------------------------------------------------[OpenVPN Server]---------------------[Internet] Physical: [Win32 Netbook]-----802.11n lite WLAN---[AP]-----100baseT-----[GigE Switch]------GigE------[pfSense 2.0.1]-------GigE-----------[CISCO WAN]The pfSense box runs in a ESX 5 VM (2 vCPU, 512MB RAM, E1000 NICs) on a Core2Duo @ 2.66GHz/8GB RAM machine.
Test Results:
Test Run A - Without openVPN, with net.inet.ip.fastforwarding = 0 (default): --------------------------------------------------------------------------- Test #1 client command line: iperf -c <firewall wlan="" ip=""> results: Interval=0.0-10.0sec, Transfer=29.8MBytes, Bandwidth=24.9Mbits/s Test #2 client command line: iperf -l 32768 -c <firewall wlan="" ip=""> results: Interval=0.0-10.0sec, Transfer=39.8MBytes, Bandwidth=33.3Mbits/s Test #3 client command line: iperf -w 32768 -l 65535 -c <firewall wlan="" ip=""> results: Interval=0.0-10.0sec, Transfer=36.6MBytes, Bandwith=30.7Mbits/s</firewall></firewall></firewall>Test Run B - With openVPN, with net.inet.ip.fastforwarding = 0 (default): ----------------------------------------------------------------------- Test #1 client command line: iperf -c <openvpn server="" ip=""> Results: Interval=0.0-10.4sec, Transfer=944KBytes, Bandwidth=743Kbits/s Test #2 client command line: iperf -l 32768 -c <openvpn server="" ip=""> Results: Interval=0.0-10.5sec, Transfer=1.88MBytes, Bandwidth=1.50Mbits/s Test #3 client command line: iperf -w 32768 -l 65535 -c <openvpn server="" ip=""> Results: Interval=0.0-10.9sec, Transfer=1.94MBytes, Bandwidth=1.50Mbits/s</openvpn></openvpn></openvpn>Test Run C - Without OpenVPN, with net.inet.ip.fastforwarding = 1: ------------------------------------------------------------------ Test #1 client command line: iperf -c <firewall wlan="" ip=""> results: Interval=0.0-10.0sec, Transfer=31.4MBytes, Bandwidth=26.3Mbits/sec Test#2 client command line: iperf -l 32768 -c <firewall wlan="" ip=""> results: Interval=0.0-10.0sec, Transfer=39.5MBytes, Bandwidth=33.1Mbits/sec Test#3: omitted.</firewall></firewall>Test Run D - With OpenVPN, with net.inet.ip.fastforwarding = 1: ----------------------------------------------------------------- Test #1 client command line: iperf -c <openvpn server="" ip=""> results: Interval=0.0-10.0sec, Transfer=16.0MBytes, Bandwidth=13.4Mbits/sec Test #2 client command line: iperf -l 32768 -c <openvpn server="" ip=""> results: Interval=0.0-10.0sec, Transfer=17.1MBytes, Bandwidth=14.0Mbits/sec Test #3: omitted.</openvpn></openvpn>Compare bandwidth: Test B.2=1.50Mbits/s vs. Test D.2=14.0Mbits/s. That's an almost tenfold increase in bandwidth!
Please note, that for all test runs via openVPN, traffic shaping w/ bandwidth limiting to 14600kbps is in place.Here's the CPU load. Time of reboot (to activate the setting) is marked with the red arrow. The reduced load is clearly visible in the graph:
Also, the time to forward packets is notably reduced:
-
Is there a downside to this "tuneup" ?
Is it safe to enable this everywhere without the risk of having to take a 10h drive ;)Any advice from pfsense devs on this ?
-
net.inet.ip.fastforwarding
When fast IP forwarding is enabled, IP packets are forwarded directly to the appropriate network interface with direct processing to completion, which greatly improves the throughput. All packets for local IP addresses, non-unicast, or with IP options are handled by the normal IP input processing path. All features of the normal (slow) IP forwarding path are supported including firewall (through pfil(9) hooks) checking, except ipsec(4) tunnel brokering. The IP fastforwarding path does not generate ICMP redirect or source quench messages. Compared to normal IP forwarding this can give a speedup of 40 to 60% in packet forwarding performance.
type: boolean, default: off
Source: http://segfault.in/2010/10/freebsd-net-inet-ip-sysctls-explained
Fastforwarding is purely an optimization path–if the packet requires features not supported by the fast path then it's processed normally. If this happens, a small performance hit is incurred.
-
fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.
-
ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.
-
ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.
Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.
-
I tried this to see if it had any impact on OpenVPN to my own network and there was a very noticeable improvement.
It also seemed that RDP to one of my Windows VMs was also a lot faster so I tried re-running the iperf test I reported here: http://forum.pfsense.org/index.php/topic,45590.0.html Same physical and virtual hosts were used and no changes to any configs - other than enabling fastforwarding.
Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved. ------------------------------------------------------------ Client connecting to 192.168.11.2, TCP port 5001 TCP window size: 63.0 KByte (default) ------------------------------------------------------------ [128] local 192.168.111.7 port 54316 connected with 192.168.11.2 port 5001 [ ID] Interval Transfer Bandwidth [128] 0.0-10.0 sec 867 MBytes 727 Mbits/secThat really surprised me! No TCP window size tweak required to achieve similar performance to that described in the other thread.
@cmb:
fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.
I'm not concerned about IPsec but I would really like to know if there are other potential risks or problems with using fastforwarding?
-
Wow, upon initial testing, I am showing the same level of performance improvement. Will be interested to know if anyone finds a downside/trade-off to this.
In the mean time, thanks so much!
-
Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.
Not true?
That's like saying having locks on your server racks is no good, because once an attacker stands in front of your server rack, they can simply smash it with a jackhammer, flood it with a firehose, set fire to it with gasoline or blow it into pieces with explosives :)
Another attack from http://insecure.org/sploits/arp.games.html:
ICMP redirects present a rather potent DoS. Unlike ARP cache entries, those
host routes won't expire with time. And of course no access to local
network is required, attack can be launched from anywhere. So if the target
system does accept ICMP redirects (and packets can actually reach it) that
system can be stopped from talking to any particular address on the net
(well, not all, but those that aren't on the same subnet with the target).
Nameservers would be an obvious target.Works remote if no filtering in place.
Besides, if you have to rely on ICMP redirect then your routing is flawed.
OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it. -
From http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002534.html:
- ip_fastforward gets its speed from processing the forwarded packet to
- completion (if_output on the other side) without any queues or netisr's.
- The receiving interface DMAs the packet into memory, the upper half of
- driver calls ip_fastforward, we do our routing table lookup and directly
- send it off to the outgoing interface which DMAs the packet to the
- network card. The only part of the packet we touch with the CPU is the
- IP header (unless there are complex firewall rules touching other parts
- of the packet, but that is up to you). We are essentially limited by bus
- bandwidth and how fast the network card/driver can set up receives and
- transmits.
- We handle basic errors, ip header errors, checksum errors,
- destination unreachable, fragmentation and fragmentation needed and
- report them via icmp to the sender.
- Else if something is not pure IPv4 unicast forwarding we fall back to
- the normal ip_input processing path. We should only be called from
- interfaces connected to the outside world.
- Firewalling is fully supported including divert, ipfw fwd and ipfilter
- ipnat and address rewrite.
- IPSEC is not supported if this host is a tunnel broker. IPSEC is
- supported for connections to/from local host.
- We try to do the least expensive (in CPU ops) checks and operations
- first to catch junk with as little overhead as possible.
- We take full advantage of hardware support for ip checksum and
- fragmentation offloading.
- We don't do ICMP redirect in the fast forwarding path. I have had my own
- cases where two core routers with Zebra routing suite would send millions
- ICMP redirects to connected hosts if the router to dest was not the default
- gateway. In one case it was filling the routing table of a host with close
- 300'000 cloned redirect entries until it ran out of kernel memory. However
- the networking code proved very robust and it didn't crash or went ill
- otherwise.
So, only IPSEC tunnel brokering does not work. IPSEC from/to host does work.
Not sure for traffic shaping. I can confirm that the limiter still works, though. -
Somehow I'd have imagined that ip_fastforward would have a meaningful impact at the hundreds of Mbit/s or Gbit/s speed range.
It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense …
-
And of course no access to local
network is required, attack can be launched from anywhere.Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.
We don't accept ICMP redirects by default btw, we will send them though.
Besides, if you have to rely on ICMP redirect then your routing is flawed.
Yes, should never have a network design that has multiple ingress/egress points on the same subnet.
OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it.
No, no use of ICMP redirects with CARP.
-
@cmb:
Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.
Back in the days (~2001) there used to be this NZ guy on Undernet, going by the nickname Venomous.. he pulled a prank once by hijacking a shoutcast radio stream using ICMP redirection - he had to, because "his" box was not local to the stream server's subnet and thus, no ARP spoofing :) Sure, what might have worked back then won't necessarily work today. But my point is, even if an ICMP redirect is supposed to contain 64 bits of the original datagram - whether that's going to be validated in any form or not, is entirely up to the receiving IP stack. No?
As for the sanity of networking stacks, past experience (teardrop, anyone?) shows that we shouldn't put too much trust into the various implementations, especially the closed source variants. I am arguing that, under normal operating conditions, a well designed network will have no legitimate use for ICMP redirect packets. So I'm just 1) advocating a "better safe than sorry" approach by filtering icmp redirects entirely and 2) suggesting that it doesn't (for a well designed network) matter too much that the fastforwarding path doesn't generate ICMP redirects.
As for the impact of ICMP source quench not being generated, the only time I've ever seen icmp source quench was back in exactly these days, when some freak hiding behind a fake shell provider with an OC-3 would hit our 128kbps frame relay with a multi-gigabit DDoS, knocking our entire ISP offline for a few hours. Needless to add that the entire flood of source quench messages the poor zyxel sdsl routers spat and choked out was -of course- totally pointless, if not absolutely counter-productive, as all the source addresses of the bad packets chocking our routers well were naturally spoofed. Oh well. Dude finally made it into the news when the FBI would shut them down for extortion.
I must admit though, that's a quite er.. historic file I linked there :)
Yet, there is a PDF which details some of the other issues regarding icmp redirection to be found at: www.cymru.com/gillsr/documents/icmp-redirects-are-bad.pdf
-
It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense …
That's exactly what I thought!
I'd still be curious to find out the root cause of the slowness :)Maybe it has to do with the fact that my pfSense -or rather the openVPN within- is not running on bare metal?
Note that for all tests without openVPN, pfSense itself performs well as expected, in respect to the bandwidth of the WLAN. The "physical" endpoints involved remain the same for all the tests (both with ovpn and without): the win32 netbook and this (virtualized) pfSense firewall. The measured bandwidth difference is however … dramatic.
Note also that whatever the cause may be, it doesn't happen/manifest on the fastforwarding path.
Another thing to note is, that if a client maxxed out @ 1.5Mbps, other clients could still get their 1.5Mbps. So the limiting effect was not cumulative.
-
Wow, upon initial testing, I am showing the same level of performance improvement. Will be interested to know if anyone finds a downside/trade-off to this.
Care to share some details about your setup? Is it also virtualized or bare-metal ?
And was your throughput prior to enabling ip_fastforward really as low as gridrun's only 1.5Mbps ?
-
After roughly one month of operating our pfSense boxen with ip.fastforwarding=1, I have not found any adverse effects and my users are quite happy, too.
Still keeping an eye though. Will let you know if something comes up.
-
Yesterday I tried the fast forwarding = 1 option.
I have a remote network that connects through a Vyatta router to an openvpn host behind the router at (main site) with the fast forwarding = 1, using an openvpn site-to-site configuration.
The openvpn setup looks like this:
remote site router (Vyatta) <–> main site (pfsense 2.0.1 w/ fast fowarding = 1 enabled) <--> openvpn server on linux host
So with my setup, pfsense is not the VPN server.
I have a remote network printer that receives print data on TCP port 9100. The print jobs are labels.
Consistently yesterday I had label print jobs not completing. For example a print job of 42 labels would occur. Only 40 labels would print.
Last night I switched back to fast forwarding = 0 and this morning the problem with printing is gone.
-
Hey guys, just so you know, I recently went through troubleshooting why my VPN connection was so slow (3Mbps on a 25Mbps link). I tried some of the recommendations in this thread and others, but nothing helped.
Turns out I set the transport to TCP (bad) when I created the server. I switched to UDP and now I get fast speeds (nearly 25Mbps).