Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Intel adapter, ixl driver and Mellanox cable

    Scheduled Pinned Locked Moved Hardware
    11 Posts 2 Posters 1.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      proxymoxy
      last edited by

      Hi, I am trying to get a link up between a Microtik switch and a Supermicro server used as a router/firewall running pfsense.

      The hardware in question is a Mellanox cable, MFA2P10-AXXX 25GbE SFP28 AOC, a Supermicro server that according to pciconf -lv is a 'Ethernet Connection X722 for 10GbE SFP+' and uses the ixl driver. On the other end is a Mikrotik CRS354-48G-4S+2Q+RM switch. The cable is connected to a SFP+ port on the mikrotik switch and on it the speed is set to 10Gb and full duplex.

      No matter what settings I try then I never get a link between the two. If i set the cable to connect between two mikrotik switches then I get a link right away so the cable is fine.

      What I read on the interwebz is that intel hardware like the X722 is really picky about what transceivers it talks to but this is on a Supermicro motherboard so I thought they perhaps might reprogram the firmware to be a bit more acceptable to other transceivers then intels. Also I read that the X722 only supports single speed transceivers but the cable is double speed (25Gb/10Gb), is this right about the X722 chipset?

      When trying to set the speed on the X722 then I always get some kind of error and trying with sysctl dev.ixl.2.supported_speeds=4 (which I think is for 10Gb) then I get an error saying it is read only. I will try to set it in /boot/loader.conf.local later today, restart it and see if that works.

      I am not that familiar with FreeBSD/pfSense so I might be missing something, but what can I do to get this fiber working between these two machines? Is this perhaps a dead quest?

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Does it even recognise the module? What speeds does it offer?

        Try running ifconfig -vvvm ixl0 at the CLI (assuming it's ixl0).

        Steve

        P 1 Reply Last reply Reply Quote 0
        • P
          proxymoxy @stephenw10
          last edited by

          @stephenw10 thanks for your reply.

          Yes it does recognize the module, here is the output

          ifconfig -vvvm ixl3
          ixl3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
          	description: Office2
          	options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
          	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
          	ether ac:1f:6b:79:85:eb
          	inet6 fe80::ae1f:6bff:fe79:85eb%ixl3 prefixlen 64 scopeid 0x8
          	media: Ethernet autoselect
          	status: no carrier
          	supported media:
          		media autoselect
          		media 10GBase-SFI
          		media 1000Base-KX
          		media 10Gbase-LR
          		media 10Gbase-SR
          		media 10Gbase-Twinax
          		media 1000baseLX
          		media 1000baseSX
          	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          	plugged: SFP/SFP+/SFP28 100G AOC or 25GAUI C2M AOC (No separable connector)
          	vendor: Mellanox PN: MFA2P10-A030 SN: MT1948FT02028 DATE: 2019-11-25
          	module temperature: 46.56 C Voltage: 3.25 Volts
          	RX: 0.80 mW (-0.94 dBm) TX: 0.72 mW (-1.38 dBm)
          
          	SFF8472 DUMP (0xA0 0..127 range):
          	03 04 23 00 00 00 00 00 08 00 00 06 FF 00 00 00 
          	00 00 1E 00 4D 65 6C 6C 61 6E 6F 78 20 20 20 20 
          	20 20 20 20 18 00 02 C9 4D 46 41 32 50 31 30 2D 
          	41 30 33 30 20 20 20 20 41 34 20 20 00 00 00 65 
          	18 3A 67 00 4D 54 31 39 34 38 46 54 30 32 30 32 
          	38 20 20 20 31 39 31 31 32 35 20 20 68 F8 08 01 
          	39 34 32 46 4D 41 32 34 32 4A 30 31 33 37 20 20 
          	00 00 00 00 00 00 00 00 00 00 01 00 20 00 00 00 
          
          
          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Hmm, well that looks good. Interesting set of media options presented.

            Note it shows zero incoming signal strength.

            First thing I would try is setting the media type to a fixed value to match the other end.

            P 1 Reply Last reply Reply Quote 0
            • P
              proxymoxy @stephenw10
              last edited by

              @stephenw10 the media types is one of the problems. With the ixl driver I can't set any media types, they cannot be changed it seems. If I try

              sysctl dev.ixl.2.supported_speeds: 4
              

              which I think is supposed to be 10Gb then I get

              sysctl: oid 'dev.ixl.2.supported_speeds' is read only
              

              The opposite end is set to 10Gb full duplex.
              I tried to use /boot/loader.conf.local to set the values and I get

              rc.bootup: The command '/sbin/ifconfig 'ixl2' media '10GBase-SFI'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device'
              ixl2: Media change is not supported.
              
              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                If they show as available mediaopts in ifconfig -m they should be selectable. They should appear in the drop-down in the GUI. Though that error seems to imply they are not. 🤔

                To restrict what speeds it can negotiate to in autoselect you should be able to use the advertise_speed sysctl. Though I don't have an ixl NIC up right now to test that with.

                P 1 Reply Last reply Reply Quote 0
                • P
                  proxymoxy @stephenw10
                  last edited by

                  @stephenw10 yes I can set the advertise_speed, but that means autonegotiation. This cable is set as 25Gb but can run as 10Gb if it disables CDR and it can only do that if you manually set the speed. So it seems like a catch-22 instance to me 🙄

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    So your options at thew switch do not include autoselect?

                    I've never used an 'optical' DAC device like that so I'm unsure how they usually appear. With a regular DAC cable though it's common to not present any media types except autoselect. And in those cases the NIC will usually only link at the native port speed, usually 10G. That might work fine here if both ends are 10G and support autoselect.

                    P 1 Reply Last reply Reply Quote 0
                    • P
                      proxymoxy @stephenw10
                      last edited by

                      @stephenw10 yes the switch can do autoselect. I have tried that and got no link between the two. The other end is a microtik switch on a sfp+ port (so max speed is 10Gb).

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Hmm, well you have any other DAC cables or SFP modules and fiber you can try?

                        P 1 Reply Last reply Reply Quote 0
                        • P
                          proxymoxy @stephenw10
                          last edited by

                          @stephenw10 unfortunately not between these two machines. One is in a locked cabin on the ground floor, the second is in a server room on the second floor. This cable was layed out some time ago by electricians that support this building and can not be moved easily.
                          I think I have tried this with a 10Gb DAC (copper) cable and got a link, I don't remember exactly. Will try that tomorrow.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.