Baffling pfSense 2.6.0 Issue (10G Performance)

anyn12

Packet loss per iperf UDP test is about 0.15% both ways.

Also, enabling flow control looked like a red herring, I'm back at a steady 4.3 Gbps. I also noticed the same thing when enabling powerD - seemed like the problem was fixed but then after a few hours the problem reappeared. Something to do with changing settings and rebooting?

I guess if you don't have any other thoughts I'll reconfigure to use the x552 for both LANs and see what happens -- I'm getting desperate!

stephenw10

Yeah, I would try to test using the x500 for both if you can.

If you simply reboot does it pass at full speed for some time?

Steve

anyn12

@stephenw10

A reboot doesn't change anything. But, I did notice with the powerD change, and then with the flow control change, throughput was restored for a short time. I've made tons of tunable changes with reboots that have had no impact on throughput.

I'll reconfigure the x500s and report back. Thanks again for helping me.

anyn12

@stephenw10

Steve - you were correct, moving both LANs to the x552s solved the problem. I now get 9.8 Gbps both ways on iperf single stream and real-world file transfers are at 10G speeds. I am very grateful for your help!

But, any guess as to why the x710 doesn't perform? I now need to use the x710 for WAN connections so I still have an interest in sorting this out.

stephenw10

Hmm, well it's almost certainly some change in the ixl driver.

What exact card are you using? How does it appear in pciconf -lv?

Are you able to test a 2.7 snapshot? It's possible this has already been solved.

Steve

anyn12

@stephenw10

In pciconf is appears as "Ethernet Controller X710 for 10GbE SFP+"

I'll give 2.7 a try and report back.

stephenw10

Do you have the actual PCI IDs shown? It could be something specific to the chip.

Cool_Corona

We see the same odd performance on several NIC and its related to cache and writes on the hardware.

anyn12

@stephenw10

Hi Steve,
Here is the full output:

ixl0@pci0:6:0:0: class=0x020000 card=0x02581374 chip=0x15728086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernet

ixl1@pci0:6:0:1: class=0x020000 card=0x00001374 chip=0x15728086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernet

anyn12

@cool_corona

Thank you -- can you point me to any online discussion/thread where this is discussed so I can follow along?