Very odd WAN connectivity issues while having 2 different gen intel NICs



  • So, I have PFS 2.2.6 running on SM C2758 mATX board and decided to add an old EXPI9402PT NIC to handle my WAN connection and then dedicate all integrated i350 ports for LAN segments.
    Everything were seemingly fine until I started testing WAN, at this point some very odd issues arose. Please, read carefully, saying "very odd", I really mean so.

    With 9402 NIC installed:

    • speedtest.net shows up to wirespeed DL (100M) and almost dead UL (test usually just "hangs" at this point)
    • a lot of IN errors on WAN, each speedtest run (DL part) gives +500-1500 ierr, oerr and collisions stays at 0
    • 10-50% packet loss on pings to gw / google
    • nothing unusual in dmesg / syslog

    And most interesting and odd part is that

    • exactly same issues with WAN being assigned to integrated i350 port while having 9402 installed (but not necessary assigned to any logic IF)
    • reboot / "assign interfaces" in cli didn't help
    • 9402 works perfectly fine being assigned to some LAN segment, iperf3 gives wirespeed with no errors and a-few-to-no retrans in both directions, so the NIC is OK
    • i350 ports are fine too, when used as LAN
    • my WAN connection itself is OK too, tested by attaching cable directly to my PC and spoofing correct MAC - no connectivity issues at all. Back to PFS on any NIC - same issues
    • finally, I pulled 9402 off and booted - WAN working fine again.

    What the fck is going on here? All NICs and WAN link itself works fine being tested alone, but WAN-on-either-nic-on-pfs is fcked up until 9402 is removed.
    Any ideas?

    To make it absolutely clear here is the simplest case:
    1. working PFS install, WAN on igb3 (integrated I350 port 4). Everything working fine
    2. Power off, install 9402, bootup, dont touch anything else => described issues on WAN
    3. [OPTIONAL] Assign em{0/1} (9402) to existing or new LAN interface => works fine, wire speed, no errors; assign to WAN => same issues
    4. Poweroff, remove 9402, bootup, dont touch anything else => WAN is fine again

    My net is NATed.
    WAN is plain ethernet (100baseTX-FD, physically 3-pairs connected) with DHCP & MAC auth. So the difference from iface POV between being assigned to LAN or WAN is:

    • having spoofed MAC on WAN / real on LAN (but tested with spoofed too - no issues)
    • 100baseTX fulldup on WAN / 1000baseT on LAN

    9402 and i350 are using different drivers

    
    ...............................
    ---> 9402PT
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xc020-0xc03f mem 0xdf160000-0xdf17ffff,0xdf140000-0xdf15ffff irq 17 at device 0.0 on pci3
    May 16 21:50:41 gateway kernel: em0: Using an MSI interrupt
    May 16 21:50:41 gateway kernel: em0: Ethernet address: xxxxxx
    May 16 21:50:41 gateway kernel: em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xc000-0xc01f mem 0xdf120000-0xdf13ffff,0xdf100000-0xdf11ffff irq 18 at device 0.1 on pci3
    May 16 21:50:41 gateway kernel: em1: Using an MSI interrupt
    May 16 21:50:41 gateway kernel: em1: Ethernet address: xxxxxx
    ..........................
    ---> 1st I350 integrated port
    May 16 21:50:41 gateway kernel: igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.4.0="">port 0xe0a0-0xe0bf mem 0xdf260000-0xdf27ffff,0xdf30c000-0xdf30ffff irq 20 at device 20.0 on pci0
    May 16 21:50:41 gateway kernel: igb0: Using MSIX interrupts with 9 vectors
    ......................</intel(r)></intel(r)></intel(r)> 
    

Log in to reply