Upgrade 2.2.3 to 2.2.4 with 10 gig nics degrades throughput
-
We setup a 2.2.3 system with two dual core Xeon procs, two Chelsio 10 gig nics. We followed the tuning guide here:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_CardsToday, we updated from 2.2.3 to 2.2.4 and noticed some odd this while testing through it with iperf:
4.29 GBPS and 4.21 GBPS throughput on 2.2.3 drops to 3.23GBPS 3.16, GBPS running the exact same tests, some ppm, etc. Same MTU (I belive 1500 for this test, waiting on the person doing the testing to verify.)One thing we noted is that CPU use is less on 2.2.4 running these tests, as is mbuf usage. But so is throughput. ~3% less for both CPU and memory.
Obviously this is not real world traffic. We have a set of Intel 10 gig cards that we tested with that were slower than the Chelsio cards on 2.2.3 with the same tune - a single stream wa fine but the Intel NICs did not scale with multiple streams. We are thinking about putting those back in and re-testing.
Is there something that might not be in the tuning guide that we need to change in 2.2.4? we checked, all the "tunes' we made came through the upgrade. We're puzzled by the loss in performance. any light on the subject would be appreciated. We are planning on setting up two HA clusters which will have lots of people sending backup traffic through
-
There were very few OS-level changes between 2.2.3 and 2.2.4, and none that would impact performance or any tuning parameters.
The Chelsio drivers do perform better than the Intel at this point. That much doesn't surprise me.
I'd suggest re-testing on 2.2.3 to compare, as I suspect something else has changed that's impacting things.
-
Thanks! Before we go back, we'll tripple check the changes made for the tunning parameters. But litterally the only thing that changed was the update from 2.2.3 to 2.2.4. All the servers are directly connected to the firewall - there are no switches, no OS or driver changes were made to the servers on either side. It's very odd. And I did confirm that all test results were with a 1500 byte MTU. There is no traffic going through this firewall that could vary - it is simply a test platform with connections to two servers out two Chelsio NIC's, so there is no way the "base" level of traffic varried between the tests. AS soon as we double check and can test I'll post again.