Intel 10Gb ix X552
-
I'm not 100% I have the screenshots matched up correctly but it looks like iperf itself is using far more CPU in pfSense than it is in FreeBSD. And that the interrupt load from the driver queues is far higher in pfSense. Would you agree?
That can be typical of a CPU running at a lower frequency. Did you check the sysctls to make sure the CPUs are running at the same speed in both cases?
The loading caused by pf appears as the interrupt load on the NIC queues. If it was not actually disabled that's where it would appear. That would also match the one direction is OK finding as pfSense allows out all traffic by default.
It's hard to believe that CPU would push close to 10Gbps with pf enabled in FreeBSD if it was actually filtering anything. Did you actually see any drop in speed or increase in CPU load when you tested that?
Interesting issue though.
Steve
-
Hi, I'm working on that problem with Belgarath.
sysctl show the same for both systems.
-
pfSense is not pushing 10GBps even with pf disabled, so that i why I am confused.
pfctl -d has no impact on the throughput on the pfsense. -
Hmm, pfctl -d will disable pf only until any change is made in the gui or the ruleset is reloaded which can be triggered by a number of things. However if will report it's already disabled if you run it again (and it is still disabled). You might try disabling it in the GUI in System > Advanced > Firewall/NAT to be sure.
It's interesting that neither box is running at 2201MHz which would be Turbo mode. But they are certainly comparable though with both set the same.
Steve
-
Hardware on both is identical, we disabled turbo mode in bios on both.
we made sure pf was disabled through the duration of the test but will check again with gui suggestion just to be sure. -
Same results when natting i disabled from GUI
-
I would recommend not using pfSense as a client or server for iperf as it does not reflect actual routing performance AT ALL. pfSense is configured to be good at routing not hosting. Both use the network but in completely different ways.
-
Try with both a client and server other than pfSense, but on different sides of the box. You'll probably have to open a firewall port for that.
-
On ubuntu and freebsd it is working with above 9Gbps both ways
-
A bit late but perhaps it helps out @belgarath.
I have an issue where PFsense on the smae hardware gets about 2-4 GB/s out of those interfaces but FreeBSD is getting 9.5 GB/salso load on the FreeBSD side is lower.
Linux and FreeBSD is not doing any NAT job and passing pf rules on top of this so it must be faster. And the
second thing is that you will be able to play around with some and/or more settings to get different numbers
of this tests. But the main and most urgent thing is here to test with NetIO or iPerf 3 through pfSense, either
from LAN port to LAN port or between the WAN and LAN ports and not on the machine itself. By the way I
really think that pfSense is not only FreeBSD plus some new GUI running like an ordinary program , it is
more then that, too many changes and other things will be turn it into its own group or level.It seems that cpu is exhausted while doing the work with PFsense, as the cpu seemed to be an issue I tried disable firewall processing on those interfaces but the results would improve by decimal parts so it does not look like it is the firewall issue.
If you are using PPPoE you will be CPU core single threaded and if not CPU multi core usage will be the result!
For sure with a pfSense version that is using all core + HT you might be able to get once more again totally other
results and numbers. Only this can be different!I tired different versions of pfsense and the results are more or less consistent, I'm getting anywhere between 1.8 and 3.5 Gbps
As normal it will be something around 2 GBit/s and 4 GBit/s as real throughput between two 10 GBit/s connections
based on the used protocols and/or used programs or offered services, but if you would see more between the test
together with iPerf you could try out to produce more streams something like 8 or 10 streams could be doing the job.General:
- HT enabling or disabling in the BIOS
- PowerD (hi adaptive, adaptive or maximum)
- Fast and enough RAM
Tunings:
Now this section can as above tried out as a single change or all together or only some combined changing´s.- mbuf size to 65.000 or to 1.000.000
Together with a broadcom NIC the 65000 was one times matching well and together with Intel NICs the 1000000 was fine - changing the entire amount of network queues from 2 to 4 (less or more try it out)
each cpu core (also the HT) is opening for each lan port one or more queues, driver pending!
You can now try out to limit or high up this numbers, that it will be matching at best to your hardware and
delivering the best results to you.
