OpenVPN performance - Huge Improvement

heper

Is there a downside to this "tuneup" ?
Is it safe to enable this everywhere without the risk of having to take a 10h drive ;)

Any advice from pfsense devs on this ?

gridrun

net.inet.ip.fastforwarding

When fast IP forwarding is enabled, IP packets are forwarded directly to the appropriate network interface with direct processing to completion, which greatly improves the throughput. All packets for local IP addresses, non-unicast, or with IP options are handled by the normal IP input processing path. All features of the normal (slow) IP forwarding path are supported including firewall (through pfil(9) hooks) checking, except ipsec(4) tunnel brokering. The IP fastforwarding path does not generate ICMP redirect or source quench messages. Compared to normal IP forwarding this can give a speedup of 40 to 60% in packet forwarding performance.

type: boolean, default: off

Source: http://segfault.in/2010/10/freebsd-net-inet-ip-sysctls-explained

Fastforwarding is purely an optimization path–if the packet requires features not supported by the fast path then it's processed normally. If this happens, a small performance hit is incurred.

cmb

fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.

gridrun

ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.

cmb

@gridrun:

ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.

Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.

biggsy

I tried this to see if it had any impact on OpenVPN to my own network and there was a very noticeable improvement.

It also seemed that RDP to one of my Windows VMs was also a lot faster so I tried re-running the iperf test I reported here: http://forum.pfsense.org/index.php/topic,45590.0.html Same physical and virtual hosts were used and no changes to any configs - other than enabling fastforwarding.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

------------------------------------------------------------
Client connecting to 192.168.11.2, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[128] local 192.168.111.7 port 54316 connected with 192.168.11.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[128]  0.0-10.0 sec   867 MBytes   727 Mbits/sec

That really surprised me! No TCP window size tweak required to achieve similar performance to that described in the other thread.

@cmb:

fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.

I'm not concerned about IPsec but I would really like to know if there are other potential risks or problems with using fastforwarding?

Helix26404

Wow, upon initial testing, I am showing the same level of performance improvement. Will be interested to know if anyone finds a downside/trade-off to this.

In the mean time, thanks so much!

gridrun

Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.

Not true?

That's like saying having locks on your server racks is no good, because once an attacker stands in front of your server rack, they can simply smash it with a jackhammer, flood it with a firehose, set fire to it with gasoline or blow it into pieces with explosives :)

Another attack from http://insecure.org/sploits/arp.games.html:

ICMP redirects present a rather potent DoS. Unlike ARP cache entries, those
host routes won't expire with time. And of course no access to local
network is required, attack can be launched from anywhere. So if the target
system does accept ICMP redirects (and packets can actually reach it) that
system can be stopped from talking to any particular address on the net
(well, not all, but those that aren't on the same subnet with the target).
Nameservers would be an obvious target.

Works remote if no filtering in place.

Besides, if you have to rely on ICMP redirect then your routing is flawed.
OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it.

gridrun

From http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002534.html:

ip_fastforward gets its speed from processing the forwarded packet to

completion (if_output on the other side) without any queues or netisr's.

The receiving interface DMAs the packet into memory, the upper half of

driver calls ip_fastforward, we do our routing table lookup and directly

send it off to the outgoing interface which DMAs the packet to the

network card. The only part of the packet we touch with the CPU is the

IP header (unless there are complex firewall rules touching other parts

of the packet, but that is up to you). We are essentially limited by bus

bandwidth and how fast the network card/driver can set up receives and

transmits.

We handle basic errors, ip header errors, checksum errors,

destination unreachable, fragmentation and fragmentation needed and

report them via icmp to the sender.

Else if something is not pure IPv4 unicast forwarding we fall back to

the normal ip_input processing path. We should only be called from

interfaces connected to the outside world.

Firewalling is fully supported including divert, ipfw fwd and ipfilter

ipnat and address rewrite.

IPSEC is not supported if this host is a tunnel broker. IPSEC is

supported for connections to/from local host.

We try to do the least expensive (in CPU ops) checks and operations

first to catch junk with as little overhead as possible.

We take full advantage of hardware support for ip checksum and

fragmentation offloading.

We don't do ICMP redirect in the fast forwarding path. I have had my own

cases where two core routers with Zebra routing suite would send millions

ICMP redirects to connected hosts if the router to dest was not the default

gateway. In one case it was filling the routing table of a host with close

300'000 cloned redirect entries until it ran out of kernel memory. However

the networking code proved very robust and it didn't crash or went ill

otherwise.

So, only IPSEC tunnel brokering does not work. IPSEC from/to host does work.
Not sure for traffic shaping. I can confirm that the limiter still works, though.

dhatz

Somehow I'd have imagined that ip_fastforward would have a meaningful impact at the hundreds of Mbit/s or Gbit/s speed range.

It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense …

cmb

@gridrun:

And of course no access to local
network is required, attack can be launched from anywhere.

Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.

We don't accept ICMP redirects by default btw, we will send them though.

@gridrun:

Besides, if you have to rely on ICMP redirect then your routing is flawed.

Yes, should never have a network design that has multiple ingress/egress points on the same subnet.

@gridrun:

OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it.

No, no use of ICMP redirects with CARP.

gridrun

@cmb:

Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.

Back in the days (~2001) there used to be this NZ guy on Undernet, going by the nickname Venomous.. he pulled a prank once by hijacking a shoutcast radio stream using ICMP redirection - he had to, because "his" box was not local to the stream server's subnet and thus, no ARP spoofing :) Sure, what might have worked back then won't necessarily work today. But my point is, even if an ICMP redirect is supposed to contain 64 bits of the original datagram - whether that's going to be validated in any form or not, is entirely up to the receiving IP stack. No?

As for the sanity of networking stacks, past experience (teardrop, anyone?) shows that we shouldn't put too much trust into the various implementations, especially the closed source variants. I am arguing that, under normal operating conditions, a well designed network will have no legitimate use for ICMP redirect packets. So I'm just 1) advocating a "better safe than sorry" approach by filtering icmp redirects entirely and 2) suggesting that it doesn't (for a well designed network) matter too much that the fastforwarding path doesn't generate ICMP redirects.

As for the impact of ICMP source quench not being generated, the only time I've ever seen icmp source quench was back in exactly these days, when some freak hiding behind a fake shell provider with an OC-3 would hit our 128kbps frame relay with a multi-gigabit DDoS, knocking our entire ISP offline for a few hours. Needless to add that the entire flood of source quench messages the poor zyxel sdsl routers spat and choked out was -of course- totally pointless, if not absolutely counter-productive, as all the source addresses of the bad packets chocking our routers well were naturally spoofed. Oh well. Dude finally made it into the news when the FBI would shut them down for extortion.

I must admit though, that's a quite er.. historic file I linked there :)

Yet, there is a PDF which details some of the other issues regarding icmp redirection to be found at: www.cymru.com/gillsr/documents/icmp-redirects-are-bad.pdf

gridrun

@dhatz:

It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense …

That's exactly what I thought!
I'd still be curious to find out the root cause of the slowness :)

Maybe it has to do with the fact that my pfSense -or rather the openVPN within- is not running on bare metal?

Note that for all tests without openVPN, pfSense itself performs well as expected, in respect to the bandwidth of the WLAN. The "physical" endpoints involved remain the same for all the tests (both with ovpn and without): the win32 netbook and this (virtualized) pfSense firewall. The measured bandwidth difference is however … dramatic.

Note also that whatever the cause may be, it doesn't happen/manifest on the fastforwarding path.

Another thing to note is, that if a client maxxed out @ 1.5Mbps, other clients could still get their 1.5Mbps. So the limiting effect was not cumulative.

dhatz

@Helix26404:

Wow, upon initial testing, I am showing the same level of performance improvement. Will be interested to know if anyone finds a downside/trade-off to this.

Care to share some details about your setup? Is it also virtualized or bare-metal ?

And was your throughput prior to enabling ip_fastforward really as low as gridrun's only 1.5Mbps ?

gridrun

After roughly one month of operating our pfSense boxen with ip.fastforwarding=1, I have not found any adverse effects and my users are quite happy, too.

Still keeping an eye though. Will let you know if something comes up.

wm408

Yesterday I tried the fast forwarding = 1 option.

I have a remote network that connects through a Vyatta router to an openvpn host behind the router at (main site) with the fast forwarding = 1, using an openvpn site-to-site configuration.

The openvpn setup looks like this:

remote site router (Vyatta) <–> main site (pfsense 2.0.1 w/ fast fowarding = 1 enabled) <--> openvpn server on linux host

So with my setup, pfsense is not the VPN server.

I have a remote network printer that receives print data on TCP port 9100. The print jobs are labels.

Consistently yesterday I had label print jobs not completing. For example a print job of 42 labels would occur. Only 40 labels would print.

Last night I switched back to fast forwarding = 0 and this morning the problem with printing is gone.

lucky

Hey guys, just so you know, I recently went through troubleshooting why my VPN connection was so slow (3Mbps on a 25Mbps link). I tried some of the recommendations in this thread and others, but nothing helped.

Turns out I set the transport to TCP (bad) when I created the server. I switched to UDP and now I get fast speeds (nearly 25Mbps).