C3K NIC with QinQ interfaces really slow upload

yswery

We have C3K NIC - ix(4) running on pfsense 2.4.4-p2

Everything works fine, BUT, we have a few QinQ interfaces set up for downstream DSL/Fibre clients and seems that they are having issues with super slow upload speeds. and by that I mean on a 100/100 connection they get 100mbps down and 0.2mbps up and constant "drop outs"

I saw there is on the https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html page a mention of thew ix(4) NICS

I see that

kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"

is already in /boot/loader.conf

This QinQ set up worked fine on our other pfsense box (with some crappy Realtek card) but with this it seems really bad and weird and inconstant.

Does anyone have the ability to point me the right direction of what is the issue and what I can try to potentially resolve this?

stephenw10

If that's the only difference there I'd look at hardware offloading. Specifically vlan tagging:

ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

re2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>

VLAN_HWSO maybe looking at what I have here.

Steve

yswery

Thanks for the reply @stephenw10 are you talking about the "Hardware Checksum Offloading" and "Hardware TCP Segmentation Offloading" in the UI? or is there something else.

My current options used

re0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>

and

ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

Edit: my NICs:

re0@pci0:2:0:0:	class=0x020000 card=0x34687470 chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

and

ix3@pci0:7:0:1:	class=0x020000 card=0x00008086 chip=0x15e48086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 1GbE'
    class      = network
    subclass   = ethernet

And Drivers loaded

grep ix3 /var/run/dmesg.boot
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xdd200000-0xdd3fffff,0xdd600000-0xdd603fff at device 0.1 on pci7
ix3: Using MSI-X interrupts with 9 vectors
ix3: Ethernet address: ac:1f:6b:b1:d8:af
ix3: netmap queues/slots: TX 8/2048, RX 8/2048```

stephenw10

You can disable/enable any of those options manually only those that have proved an issue in the past are in the GUI.

However it looks like you have LRO enabled on the ix NIC and that's something we disable by defaullt. I have it disabled there on the 5100, which is C3K. That is a GUI option so disable that first.

Steve