Intel X710-DA2 and pfSense 2.6 gives NO CARRIER

Dobby_

What cables and optics you are using? DAC or fibre optic modules? What vendor? Active or passive?

Does anyone have any ideas as to a perm solution for this?

What is the other side on the fibre cable? Looks like a auto negotiation mismatch at start up, for me. But many others were also running into similar issues like you and the most where based on the modules and cables it self or using DAC
instead of fibre optics and cables.

echel0n

@dobby_
card optics: Intel E10GSFPSRX Compatible 10GBASE-SR SFP+ 850nm 300m DOM Duplex LC MMF Optical Transceiver Module from FS.com
switch optics: Arista Networks SFP-10G-SR Compatible 10GBASE-SR SFP+ 850nm 300m DOM Duplex LC MMF Optical Transceiver Module from FS.com
cables: 0.5m (1.6ft) LC UPC to LC UPC Duplex OM3 Multimode PVC (OFNR) 2.0mm Fiber Optic Patch Cable from FS.com

I use these same items in our other systems that are Linux-based without issues, the only system we have that has issue is our pfSense systems which are BSD-based

stephenw10

What does the link come up as after you replug the module?

What does ifconfig -vvvm ixl0 show?

echel0n

@stephenw10 below is output for ixl1 from ifconfig after a replug

ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 3c:fd:fe:02:1c:a2
        inet6 fe80::3efd:feff:fe02:1ca2%ixl1 prefixlen 64 scopeid 0x2
        inet 172.16.1.2 netmask 0xfffffff8 broadcast 172.16.1.7
        inet 172.16.1.1 netmask 0xfffffff8 broadcast 172.16.1.7 vhid 2
        carp: MASTER vhid 2 advbase 1 advskew 0
        media: Ethernet autoselect (10Gbase-SR <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 10GBase-SFI
                media 10Gbase-LR
                media 10Gbase-SR
                media 10Gbase-Twinax
                media 1000baseLX
                media 1000baseSX
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: G2130658304 DATE: 2021-12-08
        module temperature: 39.51 C Voltage: 3.33 Volts
        RX: 0.61 mW (-2.14 dBm) TX: 0.57 mW (-2.39 dBm)

        SFF8472 DUMP (0xA0 0..127 range):
        03 04 07 10 00 00 00 00 00 00 00 06 67 02 00 00
        08 03 00 1E 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 53
        52 2D 38 35 20 20 20 20 41 20 20 20 03 52 00 BE
        00 3A 00 00 47 32 31 33 30 36 35 38 33 30 34 20
        20 20 20 20 32 31 31 32 30 38 20 20 68 FA 03 F4
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

stephenw10

@echel0n said in Intel X710-DA2 and pfSense 2.6 gives NO CARRIER:

SFP-10GSR-85

Hmm, and it shows up the same after a reboot, it just fails to link? Does it show incoming signal at least?

biggsy

I have next to no experience with CARP but could this be significant?

carp: MASTER vhid 2 advbase 1 advskew 0

echel0n

So I replaced both SFP modules on the x710-da2 nic, power off/on testing appeared good, however, leaving it unplugged for a prolonged time (10 - 20 minutes) and then powering back on, ixl0 came up but ixl1 did not, then checking again I found ixl0 lost carrier again, I bounced the ports and both worked again, I'm not sold on this being a SFP module or cable related issue anymore, I may have to learn more towards CARP possibly, anyone else have thoughts?

stephenw10

CARP should have nothing to do with the link. That CARP line you see looks correct assuming it's the master node.

echel0n

OK, so what I did was is added a 3rd dedicated interface just for SYNC. bounced power to firewall #2 and this time only ixl0 was "no carrier" but ixl1 was good. I did the same thing for firewall #1 and both ixl0 and ixl1 had carrier, so this issue seems very random, also I checked the logs on the Arista switch and did not see anything indicating the port was shutdown do to a loop, I'm baffled on this, not sure what the issue is anymore, FreeBSD on our FreeNAS box works perfectly with the same SFP modules ...

stephenw10

@echel0n said in Intel X710-DA2 and pfSense 2.6 gives NO CARRIER:

FreeBSD on our FreeNAS box works perfectly with the same SFP modules ...

But a different NIC?

There might be some link error data in the MAC stats in the sysctls.

You might be able to set the advertised link speeds to only what you need.

Or try setting a fixed link speed at both ends.

Steve

echel0n

Yes, our FreeNAS uses a X520-DA2, I will try hard-coding the speed/duplex on both the pfSense box and switch to see what comes of it and reply back after, thanks!