10Gbps - pfSense 3,4Gbps / ubuntu 9,4Gbps ??

[hege]

Hi,

I only get arround 3,4Gbps with my setup, (only 4,4Gbps with pctl -d / pfSense 2.2.5)
With Ubuntu 14.04 I get 9,4Gbps

Setup: (both systems are 1:1)
CPU: Intel i5-4590 @3.3Ghz
Ram: 2x 8Gb
HDD: 120 Gb SSD
NIC: Intel  X520-DA2

PC1 <- X520-DA2 -> PC2

used commands:
Server: iperf -s
Client: iperf -c SERVER -t 10

Changed settings:
/boot/loader.conf
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"

sysctl hw.intr_storm_threshold=10000

What did I do wrong, what have I forgotten?

[Harvy66]

Just making sure I got this. You've connected two systems directly together via Intel 520 10Gb NICs, and when you use Ubuntu, you get about 9Gb, and when you use PFSense, you get about 3Gb/s?

What version of IPerf are you using on Linux? PFSense is only 2.x while 3.x now exists.

I also noticed your Ubuntu boxes are defaulting with larger TCP windows.

[hege]

@Harvy66:

You've connected two systems directly together via Intel 520 10Gb NICs, and when you use Ubuntu, you get about 9Gb, and when you use PFSense, you get about 3Gb/s?

Exactly!

@Harvy66:

What version of IPerf are you using on Linux? PFSense is only 2.x while 3.x now exists.
I also noticed your Ubuntu boxes are defaulting with larger TCP windows.

Ubuntu 14.04 has iperf 2.0.5 (same as pfSense)
I just tried it with same TCP Window but it makes no difference :(

[q54e3w]

I found FreeBSD needs more tuning to get wire speeds through Intels 10gbe NIC's. Its beyond just TCP window size, its worth reading the Intel manual and some of the other manufacturers tuning guides (Mellanox etc) which will give you pointers. I''m not sure theres a one size fits all fix sadly and you may at some point run into physical hardware limits such as CPU performance to fill the pipe although I suspect your i5 will be fine in that regard. I also heard the Chelsio cards are more plug n play which if time is tight might be an easier route to go if you need the perf. Hang in there, the x520 can fly when correctly configured.

[cmb]

You probably don't have a firewall enabled on Ubuntu, and certainly not one that has scrub and a number of rules loaded. Disable the packet filter under System>Advanced and you'll get an equivalent test.

Things aren't really tuned for traffic to/from the firewall itself, traffic through the system is a much better test for evaluating things that matter in a firewall scenario.

[hege]

As far as I know - "pctl -d" is exactly what the GUI checkbox does and I only got 4.4Gbps - still much lower than with Ubuntu
It seems that I have to use two more PCs for a correct Test.

My initial goal was to check how much throughput I can get with two C2558 as FW CPUs,
but I only got 1.6Gbps single tcp connection with the Setup PC1 -> FW1 <-> FW2 -> PC2, so I changed to PC1 <-> PC2

@irj972
Thank you, I will check that.

[savago]

If you want to use full 10Gb/s link capacity,you must use clear freebsd and ipfw/netmap :)