Baffling pfSense 2.6.0 Issue (10G Performance)
-
Hi Stephen,
I think your guess on the x710 may have been right. I enable RX flow control on the x710, per this thread: https://forum.netgate.com/topic/162333/intel-x710-issues.After enabling flow control, I can get full 10Gbps throughput on iperf from LAN1 (x552) to LAN2 (x710). Real world file transfer files from LAN1 to LAN2 are about 1 GB/s.
But, it is intermittent: every few seconds during iperf test I get a drop back down to 4.3 Gbps, and sometimes the throughput drops back to 4.3 Gbps for an extended time. I've attached an iperf graph.
Do you have any thoughts on what could be happening here?
-
Hmm, well often that is some sort of TCP windowing issue. However that is negotiated between the endpoints and that, I assume, has not changed since you were running 2.5.2?
-
The endpoint configurations have not changed since 2.5.2. And, the endpoints get 10Gbps both directions when on same segment. Do you think the fact that iperf between pfsense and endpoints shows 9.7Gbps from pfsense to endpoints but ~2 Gbps from endpoints to pfsense is an indicator of anything?
I'm not sure if this is helpful, but UDP tests with iperf show 10Gbps in both directions.
-
It shows that pfSense is a bad TCP endpoint but that is known. It's configured to be a router not a server. Testing to or from pfSense directly always gives poor results. It's a useful test to prove it's linked at >1Gbps but not much else.
It also shows the receive side is significantly more processor intensive.The UDP test shows zero packet loss at 10G line rate?
Steve
-
Packet loss per iperf UDP test is about 0.15% both ways.
Also, enabling flow control looked like a red herring, I'm back at a steady 4.3 Gbps. I also noticed the same thing when enabling powerD - seemed like the problem was fixed but then after a few hours the problem reappeared. Something to do with changing settings and rebooting?
I guess if you don't have any other thoughts I'll reconfigure to use the x552 for both LANs and see what happens -- I'm getting desperate!
-
Yeah, I would try to test using the x500 for both if you can.
If you simply reboot does it pass at full speed for some time?
Steve
-
A reboot doesn't change anything. But, I did notice with the powerD change, and then with the flow control change, throughput was restored for a short time. I've made tons of tunable changes with reboots that have had no impact on throughput.
I'll reconfigure the x500s and report back. Thanks again for helping me.
-
Steve - you were correct, moving both LANs to the x552s solved the problem. I now get 9.8 Gbps both ways on iperf single stream and real-world file transfers are at 10G speeds. I am very grateful for your help!
But, any guess as to why the x710 doesn't perform? I now need to use the x710 for WAN connections so I still have an interest in sorting this out.
-
Hmm, well it's almost certainly some change in the ixl driver.
What exact card are you using? How does it appear in
pciconf -lv?Are you able to test a 2.7 snapshot? It's possible this has already been solved.
Steve
-
In pciconf is appears as "Ethernet Controller X710 for 10GbE SFP+"
I'll give 2.7 a try and report back.
-
Do you have the actual PCI IDs shown? It could be something specific to the chip.
-
We see the same odd performance on several NIC and its related to cache and writes on the hardware.
-
Hi Steve,
Here is the full output:ixl0@pci0:6:0:0: class=0x020000 card=0x02581374 chip=0x15728086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernetixl1@pci0:6:0:1: class=0x020000 card=0x00001374 chip=0x15728086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernet -
Thank you -- can you point me to any online discussion/thread where this is discussed so I can follow along?
