Net.inet.ip.fastforwarding performance tweak

[gridrun]

A while ago, I found that enabling net.inet.ip.fastforwarding would greatly aid with openVPN throughput of a pfSense virtual machine.
As a side effect, the setting also lead to a significant decrease in CPU load. Detailed info over here: http://forum.pfsense.org/index.php/topic,47567.0.html

I can now confirm that the setting also reduces CPU load on ALIX based pfSenses:

Didn't bother to mark the point when I enabled the setting, as it's pretty obvious from the above graphs.
So far, I discovered no ill side effects on the ALIX hardware either. One pfSense user however reported fastforwarding related difficulties with remote printing.
Some descriptive information about net.inet.ip.fastforwarding can be found in: http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002534.html

I have no data on openVPN performance so far, because I don't use that on the Alix here.

[stephenw10]

Wow, I had not read that thread (or maybe I forgot about it  ::)). That's a massive performance increase, makes me want to run some tests immediately.

Steve

[gridrun]

Yes, please do and report back your findings :)

[Nachtfalke]

Will this improve performance on normal routing ?
Are there any side effects ?
What is this option doing exactly ? Anything to read about that ?

[cmb]

It'll break IPsec for sure, and potentially other things, which is why we don't enable it by default.

[dhatz]

@gridrun:

A while ago, I found that enabling net.inet.ip.fastforwarding would greatly aid with openVPN throughput of a pfSense virtual machine.

I wonder if it's still a problem with the newer FreeBSD 8.3 kernel used by pfsense 2.1-BETA

I would also imagine that FreeBSD kernel developers might be interested to look into this, as both OpenVPN and VMware are very widely deployed.

[stephenw10]

Reporting back. I ran some tests on my home box with iperf. The box is a Pentium4-M underclocked to 1.2GHz via a low FSB. Not a fast machine!

I tested first using the iperf server in the pfSense box.
No fast fowarding:

C:\Temp>iperf -v
iperf version 1.7.0 (13 Mar 2003) win32 threads

C:\Temp>iperf -c 192.168.2.1 -t 30
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1912] local 192.168.2.22 port 1907 connected with 192.168.2.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0-30.0 sec  1.07 GBytes   306 Mbits/sec

With fast forwarding:

C:\Temp>iperf -c 192.168.2.1 -t 30
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1912] local 192.168.2.22 port 2127 connected with 192.168.2.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0-30.0 sec  1.04 GBytes   297 Mbits/sec

No difference to speak of, within normal test variation.
Then I tested through the box running the iperf server on a different pfSense box.
No fast fowarding:

C:\Temp>iperf -c 192.168.1.129 -t 30
------------------------------------------------------------
Client connecting to 192.168.1.129, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1912] local 192.168.2.22 port 1966 connected with 192.168.1.129 port 5001
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0-30.0 sec   672 MBytes   188 Mbits/sec

With fast forwarding:

C:\Temp>iperf -c 192.168.1.129 -t 30
------------------------------------------------------------
Client connecting to 192.168.1.129, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1912] local 192.168.2.22 port 2126 connected with 192.168.1.129 port 5001
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0-30.0 sec   858 MBytes   240 Mbits/sec

20% improvement.  :)
Not quite the dramatic increase you saw but I suspect you may have had other issues since 1.5Mbps seems too slow.
I also note that biggsy's result is tainted by the fact that his initial test was using an 8k window size but the improved result used a 63k window size. The different versions/ports of iperf seem to have different defaults which can catch you out.  ;)

Since this is my home box I'll see if anything has broken over the next few days.

Steve

[gridrun]

@dhatz:

I wonder if it's still a problem with the newer FreeBSD 8.3 kernel used by pfsense 2.1-BETA

I have since sold the small WISP business that used the virtualized pfSense system I reported on earlier. However, my new employer (an ISP) asked me to build a vSphere cluster and help them virtualize large parts of their central office. So I will soon be able to do further testing with pfSense in this virtual environment.