C3K NIC with QinQ interfaces really slow upload



  • We have C3K NIC - ix(4) running on pfsense 2.4.4-p2

    Everything works fine, BUT, we have a few QinQ interfaces set up for downstream DSL/Fibre clients and seems that they are having issues with super slow upload speeds. and by that I mean on a 100/100 connection they get 100mbps down and 0.2mbps up and constant "drop outs"

    I saw there is on the https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html page a mention of thew ix(4) NICS

    I see that

    kern.ipc.nmbclusters="1000000"
    kern.ipc.nmbjumbop="524288"
    

    is already in /boot/loader.conf

    This QinQ set up worked fine on our other pfsense box (with some crappy Realtek card) but with this it seems really bad and weird and inconstant.

    Does anyone have the ability to point me the right direction of what is the issue and what I can try to potentially resolve this?


  • Netgate Administrator

    If that's the only difference there I'd look at hardware offloading. Specifically vlan tagging:

    ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    
    re2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    

    VLAN_HWSO maybe looking at what I have here.

    Steve



  • Thanks for the reply @stephenw10 are you talking about the "Hardware Checksum Offloading" and "Hardware TCP Segmentation Offloading" in the UI? or is there something else.

    My current options used

    re0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
    	options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    

    and

    ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    

    Edit: my NICs:

    re0@pci0:2:0:0:	class=0x020000 card=0x34687470 chip=0x816810ec rev=0x06 hdr=0x00
        vendor     = 'Realtek Semiconductor Co., Ltd.'
        device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
        class      = network
        subclass   = ethernet
    

    and

    ix3@pci0:7:0:1:	class=0x020000 card=0x00008086 chip=0x15e48086 rev=0x11 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = 'Ethernet Connection X553 1GbE'
        class      = network
        subclass   = ethernet
    

    And Drivers loaded

    grep ix3 /var/run/dmesg.boot
    ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xdd200000-0xdd3fffff,0xdd600000-0xdd603fff at device 0.1 on pci7
    ix3: Using MSI-X interrupts with 9 vectors
    ix3: Ethernet address: ac:1f:6b:b1:d8:af
    ix3: netmap queues/slots: TX 8/2048, RX 8/2048```

  • Netgate Administrator

    You can disable/enable any of those options manually only those that have proved an issue in the past are in the GUI.

    However it looks like you have LRO enabled on the ix NIC and that's something we disable by defaullt. I have it disabled there on the 5100, which is C3K. That is a GUI option so disable that first.

    Steve


Log in to reply