10Gbe Tuning?

jasonlitka

The newest builds of 2.1.1, the ones where the newer drivers are back in, have gotten me to about double what I was seeing when I use multiple threads.

Likely: It's as good as it's going to get prior to 2.2.

jasonlitka

@gonzopancho:

Likely: It's as good as it's going to get prior to 2.2.

I can live with that for now. If FreeBSD 10 does to pfSense what FreeBSD 9 did for FreeNAS, I should be able to hit wire speed once 2.2 drops.

brunoc

@gonzopancho:

Likely: It's as good as it's going to get prior to 2.2.

So, to confirm, does that mean you guys know the source of the poor throughput and that will be addressed in 2.2 (either a fix or due to the upgrade to freeBSD 10)?

Thanks.

FreeBSD doesn't go wirespeed on 10G NICs (without using large frames or tricks like netmap).
Neither does linux.

the intel 10G driver(s) are good, but not great.

that all said, the situation should improve with 2.2

jasonlitka

@gonzopancho:

FreeBSD doesn't go wirespeed on 10G NICs (without using large frames or tricks like netmap).
Neither does linux.

the intel 10G driver(s) are good, but not great.

that all said, the situation should improve with 2.2

I was able to get ~8Gbit/s between two FreeNAS 9.x boxes without jumbo frames when using 4 threads. That's pretty close to wire.

brunoc

What will change in 2.2 that is expected to improve things for the 10G Intel cards? Would it be an upgrade to FreeBSD 10 or driver/tuning updates?

For the current pfSense version (2.1.x) would Myricom 10-Gigabit Ethernet (Myri10GE) cards perform better (10Gbps speeds)?

Thanks.

@Jason:

I was able to get ~8Gbit/s between two FreeNAS 9.x boxes without jumbo frames when using 4 threads. That's pretty close to wire.

OK, Jason… FreeBSD won't forward at wirespeed on 10Gbps networks.

Since the BSDRP guy can only manage to forward (no firewall, just fast forwarding) at a pinch over 1.8Mpps, (and you were doing, by my best estimate, 5.5Mpps), I'm going to assert that we still have work to do.

brunoc: we're currently engaged in a 10G performance study, but yes, part of the solution will be tuning, and part of it will be the threaded pf in pfSense version 2.2.

jasonlitka

@gonzopancho:

@Jason:

I was able to get ~8Gbit/s between two FreeNAS 9.x boxes without jumbo frames when using 4 threads. That's pretty close to wire.

OK, Jason… FreeBSD won't forward at wirespeed on 10Gbps networks.

Since the BSDRP guy can only manage to forward (no firewall, just fast forwarding) at a pinch over 1.8Mpps, (and you were doing, by my best estimate, 5.5Mpps), I'm going to assert that we still have work to do.

brunoc: we're currently engaged in a 10G performance study, but yes, part of the solution will be tuning, and part of it will be the threaded pf in pfSense version 2.2.

One interesting thing of note is that at least one user here has had a lot of luck using pfSense on vSphere. With virtualized NICs he seems to be getting better throughput than I am on bare-metal, even though I'm using faster CPUs, so I'm wondering how much of this is the Intel drivers. The newest ones are better than the last, but they're still not exactly screaming along.

I'll keep an eye on the 2.2 section of the forums. Once it gets stable enough to run as the backup of a CARP pair (next to a 2.1.x box) maybe I'll upgrade one system at the office for testing.

If there's any tuning that you want me to test out that can be done on 2.1.x, let me know. I'd be glad to try a few things on my boxes.

tojaktoty

@Jason:

One interesting thing of note is that at least one user here has had a lot of luck using pfSense on vSphere. With virtualized NICs he seems to be getting better throughput than I am on bare-metal, even though I'm using faster CPUs, so I'm wondering how much of this is the Intel drivers. The newest ones are better than the last, but they're still not exactly screaming along.

Where was that discussion about pfsense on esxi providing more throughput than your similar bare metal. I looked and can't find it. Thanks

stephenw10

I think it was this thread. I remember this figure seeming surprisingly high at the time, it still does:
https://forum.pfsense.org/index.php?topic=72142.msg395165#msg395165

Steve

I can't imagine any real performance gain for pf when running under VMware.

bl0815

I have pfsense 2.1.4 on a new box with two CPUs: E5-2667 @2.90GHz, all 12 cores enabled, but hyperthreading and vt disabled.
All traffic goes over one intel x520-sr2.
With my simple test setup ( iperf between two VMs, traffic goes through the whole datacenter, with the pfsense box in the middle), I got up to 3Gbit/s (perhaps I could get more with better VMware-infrastructure) with a CPU load below 2.

my /boot/loader.conf.local:

kern.ipc.nmbclusters="262144"
kern.ipc.nmbjumbop="262144"
net.isr.bindthreads=0
net.isr.maxthreads=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.point_to_point=0
kern.random.sys.harvest.interrupt=0
net.isr.defaultqlimit=2048
net.isr.maxqlimit=40960

and my changes in system-tunables:

hw.intr_storm_threshold=10000
kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216   
net.inet.ip.fastforwarding=1
net.inet.tcp.sendbuf_inc=262144
net.inet.tcp.recvbuf_inc=262144 
net.route.netisr_maxqlen=2048
net.inet6.ip6.redirect=0
net.inet.ip.redirect=0
net.inet.ip.intr_queue_maxlen=2048

And make sure to switch off LRO and TSO of the ix-interfaces. TSO is broken with IPv6, if it is enabled, only one paket is sent at once and then the box waits for the ACK until it sends the next one…
Some of the options I found in the freebsd-wiki: https://wiki.freebsd.org/NetworkPerformanceTuning

q54e3w

Mine throughput completely sucks right now….Im seeing 600mbps (you read it right, not even 1gig) when testing iperf from my desktop to my pfSense router. Ive applied the calomel tricks and tips re buffers etc and still seeing sucky perf so I need to do some debugging for sure. Im dreaming of the lefty heights of a 2gig connection right now!

BTW, this guy nails 9.x gbps > https://forum.pfsense.org/index.php?topic=77144.msg435304#msg435304

FYI Im using a1srm 2758f board with intel x520 SFP+ optical cables etc. I'm still limited to 600mbps on a gigabit ethernet cat6 wire to my quad i350 too.

demco

Just an observation on the 9.22Gbps test result.

1. The measurement is taken on LAN interface which is a bridge of 4 10Gbps + 1 1Gbps interfaces. It would be measuring the sum of all 5 interfaces.

2. The test setup seem to be connect 1 host to each of the 10Gbps ports. Have these 4 hosts ran iperf.

3. Most report seeing around 2Gbps on 10Gbps interfaces. So 4x 2Gbps is within reach of the result.

4. If the 10Gbps ports are doing line rate, shouldn't the test be measuring 40Gbps instead of 9Gbps? Still 9Gbps is impressive on older hardware.

q54e3w

Yes, the LAN reports the traffic on the bridge (mine is setup like this also) but Id assumed he was reporting line rate on 1 port rather than (4 * 2g + 1 * 1g) speeds. You are right though, without seeing his other ports there is ambiguity. I'd assumed given he spent the time to post he had close to line rate out of 1 port which theoretically should be possible, rather than close to line rate from 4+1…. good spot.

stephenw10

You must be hitting some limit. Are the NICs connecting at 10Gbps? Are you seeing errors on the interface? What does your CPU usage look like? Large interrupt load?

Steve

Harvy66

@irj972:

Mine throughput completely sucks right now….Im seeing 600mbps (you read it right, not even 1gig) when testing iperf from my desktop to my pfSense router. Ive applied the calomel tricks and tips re buffers etc and still seeing sucky perf so I need to do some debugging for sure. Im dreaming of the lefty heights of a 2gig connection right now!

BTW, this guy nails 9.x gbps > https://forum.pfsense.org/index.php?topic=77144.msg435304#msg435304

FYI Im using a1srm 2758f board with intel x520 SFP+ optical cables etc. I'm still limited to 600mbps on a gigabit ethernet cat6 wire to my quad i350 too.

PFSense 2.2 will have better multi-core multi-stream performance. Your Atom CPU has poor single thread performance, even thought it should have decent aggregate throughput.

I'm getting 980mb, ~1.5gb with bi-directional test, with Iperf through PFSense NAT. All with 7.7% cpu load and no tweaking. The performance is entirely limited by my 2 testing computer's integrated NICs.

stephenw10

It still has almost double the single thread rating of, say, a D525 which can itself manage close to 600Mbps throughput. :-
This test used the pfSense box as the end point though so they are not comparable.

Steve

dmitripr

@stephenw10:

It still has almost double the single thread rating of, say, a D525 which can itself manage close to 600Mbps throughput. :-
This test used the pfSense box as the end point though so they are not comparable.

Steve

Steve, did you get anywhere with this?

I also just ran some iperf test, I have Atom D2550, and it's also maxing out at ~450-500 mbps when I do UDP from my pfsense box. I see the CPU staying right at 25-27% load during tests. I'm thinking that this is getting limited by the single thread of iperf on Atom.

Interestingly enough. I got a Lenovo T440 laptop with Win7, when I also run the UDP test from that (Intel NIC) it's also maxing out at 450-500 mbps.

I'm not sure what to make of that. Maybe an issue with 2.0.x iperf?

-Dmitri