Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    8200/6100 Intel x553 NIC and certain SFP+ modules

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    25 Posts 3 Posters 1.5k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S Offline
      stephenw10 Netgate Administrator
      last edited by

      Interesting. Then check the stats in the sysctl to see what sort of errors they are: sysctl dev.ix.0

      The C3K NICs are limited in the information they can get from modules about the link state. That can mean they negotiate incorrectly or fail to link with some media types. XPON modules that that hide the real link and present to the NIC as something else so there are a few things there that could cause problems.

      A 1 Reply Last reply Reply Quote 0
      • A Offline
        aholmes5 @stephenw10
        last edited by aholmes5

        @stephenw10 SFP in 8200.txt SFP in switch.txt

        Here is what I have. I don't know how to interpret this. Lots of errors:
        dev.ix.0.mac_stats.rec_len_errs: 20
        dev.ix.0.mac_stats.remote_faults: 0
        dev.ix.0.mac_stats.local_faults: 53
        dev.ix.0.mac_stats.short_discards: 7
        dev.ix.0.mac_stats.byte_errs: 1425
        dev.ix.0.mac_stats.ill_errs: 873
        dev.ix.0.mac_stats.crc_errs: 5468
        dev.ix.0.mac_stats.rx_errs: 6374

        1 Reply Last reply Reply Quote 0
        • stephenw10S Offline
          stephenw10 Netgate Administrator
          last edited by

          Mmm, interesting no one type....

          Check netstat -m make sure there's no buffer exhaustion happening.

          A 1 Reply Last reply Reply Quote 0
          • A Offline
            aholmes5 @stephenw10
            last edited by

            @stephenw10
            58799/4456/63255 mbufs in use (current/cache/total)
            49273/3559/52832/1000000 mbuf clusters in use (current/cache/total/max)
            3/2029 mbuf+clusters out of packet secondary zone in use (current/cache)
            0/1524/1524/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
            0/0/0/524288 9k jumbo clusters in use (current/cache/total/max)
            0/0/0/84161 16k jumbo clusters in use (current/cache/total/max)
            113245K/14328K/127573K bytes allocated to network (current/cache/total)
            0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
            0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
            0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
            0/0/0 requests for jumbo clusters denied (4k/9k/16k)
            0 sendfile syscalls
            0 sendfile syscalls completed without I/O request
            0 requests for I/O initiated by sendfile
            0 pages read by sendfile as part of a request
            0 pages were valid at time of a sendfile request
            0 pages were valid and substituted to bogus page
            0 pages were requested for read ahead by applications
            0 pages were read ahead by sendfile
            0 times sendfile encountered an already busy page
            0 requests for sfbufs denied
            0 requests for sfbufs delayed

            1 Reply Last reply Reply Quote 0
            • L Offline
              lio1503
              last edited by

              Hi @aholmes5

              I have the same issue. I connect my Fiber WAN on SFP+ 10G on netgate 6100 with a 10GTEK sfp+10G and my unifi switch on the second WAN on SFP+ 10G on netgate 6100 with DAC 10G cable but my download speed is reduce of 80% but my upload is fine.
              Do you found a solution ?

              1 Reply Last reply Reply Quote 0
              • stephenw10S Offline
                stephenw10 Netgate Administrator
                last edited by

                Are you seeing errors on the interfaces?

                Do you have any traffic shaping?

                L 1 Reply Last reply Reply Quote 0
                • L Offline
                  lio1503 @stephenw10
                  last edited by lio1503

                  @stephenw10
                  I see some error on interfaces and no traffic shaping configured.

                  1 Reply Last reply Reply Quote 0
                  • L Offline
                    lio1503
                    last edited by

                    Screenshot 2025-08-06 203441.png

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S Offline
                      stephenw10 Netgate Administrator
                      last edited by

                      No errors shown there. So some on the WAN?

                      How are you testing the throughput?

                      L 1 Reply Last reply Reply Quote 0
                      • L Offline
                        lio1503 @stephenw10
                        last edited by

                        @stephenw10

                        I put the wrong image sorry

                        netgate1.png netgate2.png netgate3.png

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S Offline
                          stephenw10 Netgate Administrator
                          last edited by

                          Hmm, not really a significant number there. Nothing that would cause throttling to that extent.

                          What about the PPPoE parent interface? Which I assume is ix0?
                          Try: netstat -i and ifconfig -v ix0

                          L 1 Reply Last reply Reply Quote 0
                          • L Offline
                            lio1503 @stephenw10
                            last edited by

                            Hi @stephenw10

                            Here the result of commands

                            Screenshot 2025-08-08 112744.png

                            Screenshot 2025-08-08 112955.png

                            Thanks

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S Offline
                              stephenw10 Netgate Administrator
                              last edited by

                              Ok huge number of input errors on ix0. That's what you'd expect for bad downloads.

                              You have an XPON module in there I assume? Can you access that to get stats?

                              You can check the sysctl output for ix0 to see what type of errors those are: sysctl dev.ix.0

                              L 1 Reply Last reply Reply Quote 0
                              • L Offline
                                lio1503 @stephenw10
                                last edited by stephenw10

                                @stephenw10

                                Yes I use this module on ix0 https://www.amazon.ca/dp/B06XG9DPJ7?th=1
                                and this cable to unifi switch for Lan on ix1 https://www.amazon.ca/dp/B01M09C9NZ?th=1

                                Result of sysctl dev.ix.0

                                dev.ix.0.wake: 0
                                dev.ix.0.dmac: 0
                                dev.ix.0.tso_tcp_flags_mask_last_segment: 3967
                                dev.ix.0.tso_tcp_flags_mask_middle_segment: 4086
                                dev.ix.0.tso_tcp_flags_mask_first_segment: 4086
                                dev.ix.0.fw_version: nvm 0.58.0 eTrack 0x8000084b
                                dev.ix.0.enable_aim: 0
                                dev.ix.0.advertise_speed: 0
                                dev.ix.0.fc: 3
                                dev.ix.0.mac_stats.tx_frames_1024_1522: 2688329
                                dev.ix.0.mac_stats.tx_frames_512_1023: 9258
                                dev.ix.0.mac_stats.tx_frames_256_511: 7406
                                dev.ix.0.mac_stats.tx_frames_128_255: 51767
                                dev.ix.0.mac_stats.tx_frames_65_127: 1044399
                                dev.ix.0.mac_stats.tx_frames_64: 2651
                                dev.ix.0.mac_stats.management_pkts_txd: 0
                                dev.ix.0.mac_stats.mcast_pkts_txd: 5
                                dev.ix.0.mac_stats.bcast_pkts_txd: 1
                                dev.ix.0.mac_stats.good_pkts_txd: 3803810
                                dev.ix.0.mac_stats.total_pkts_txd: 3803810
                                dev.ix.0.mac_stats.good_octets_txd: 4177580394
                                dev.ix.0.mac_stats.checksum_errs: 0
                                dev.ix.0.mac_stats.management_pkts_drpd: 0
                                dev.ix.0.mac_stats.management_pkts_rcvd: 0
                                dev.ix.0.mac_stats.recv_jabberd: 1228
                                dev.ix.0.mac_stats.recv_oversized: 0
                                dev.ix.0.mac_stats.recv_fragmented: 0
                                dev.ix.0.mac_stats.recv_undersized: 0
                                dev.ix.0.mac_stats.rx_frames_1024_1522: 2492339
                                dev.ix.0.mac_stats.rx_frames_512_1023: 8772
                                dev.ix.0.mac_stats.rx_frames_256_511: 8173
                                dev.ix.0.mac_stats.rx_frames_128_255: 30310
                                dev.ix.0.mac_stats.rx_frames_65_127: 620807
                                dev.ix.0.mac_stats.rx_frames_64: 2487
                                dev.ix.0.mac_stats.bcast_pkts_rcvd: 247
                                dev.ix.0.mac_stats.mcast_pkts_rcvd: 10
                                dev.ix.0.mac_stats.good_pkts_rcvd: 3162888
                                dev.ix.0.mac_stats.total_pkts_rcvd: 3260844
                                dev.ix.0.mac_stats.good_octets_rcvd: 3788604566
                                dev.ix.0.mac_stats.total_octets_rcvd: 3930535548
                                dev.ix.0.mac_stats.xoff_recvd: 0
                                dev.ix.0.mac_stats.xoff_txd: 0
                                dev.ix.0.mac_stats.xon_recvd: 0
                                dev.ix.0.mac_stats.xon_txd: 0
                                dev.ix.0.mac_stats.rx_missed_packets: 0
                                dev.ix.0.mac_stats.rec_len_errs: 0
                                dev.ix.0.mac_stats.remote_faults: 0
                                dev.ix.0.mac_stats.local_faults: 1
                                dev.ix.0.mac_stats.short_discards: 7
                                dev.ix.0.mac_stats.byte_errs: 9220
                                dev.ix.0.mac_stats.ill_errs: 1617
                                dev.ix.0.mac_stats.crc_errs: 87158
                                dev.ix.0.mac_stats.rx_errs: 90003
                                dev.ix.0.queue3.rx_discarded: 0
                                dev.ix.0.queue3.rx_copies: 0
                                dev.ix.0.queue3.rx_bytes: 25480
                                dev.ix.0.queue3.rx_packets: 245
                                dev.ix.0.queue3.rxd_tail: 244
                                dev.ix.0.queue3.rxd_head: 245
                                dev.ix.0.queue3.irqs: 482085
                                dev.ix.0.queue3.interrupt_rate: 31250
                                dev.ix.0.queue3.tx_packets: 1027008
                                dev.ix.0.queue3.tso_tx: 0
                                dev.ix.0.queue3.txd_tail: 1859
                                dev.ix.0.queue3.txd_head: 1859
                                dev.ix.0.queue2.rx_discarded: 0
                                dev.ix.0.queue2.rx_copies: 0
                                dev.ix.0.queue2.rx_bytes: 0
                                dev.ix.0.queue2.rx_packets: 0
                                dev.ix.0.queue2.rxd_tail: 128
                                dev.ix.0.queue2.rxd_head: 0
                                dev.ix.0.queue2.irqs: 594318
                                dev.ix.0.queue2.interrupt_rate: 31250
                                dev.ix.0.queue2.tx_packets: 1189658
                                dev.ix.0.queue2.tso_tx: 0
                                dev.ix.0.queue2.txd_tail: 1831
                                dev.ix.0.queue2.txd_head: 1831
                                dev.ix.0.queue1.rx_discarded: 0
                                dev.ix.0.queue1.rx_copies: 0
                                dev.ix.0.queue1.rx_bytes: 0
                                dev.ix.0.queue1.rx_packets: 0
                                dev.ix.0.queue1.rxd_tail: 128
                                dev.ix.0.queue1.rxd_head: 0
                                dev.ix.0.queue1.irqs: 414076
                                dev.ix.0.queue1.interrupt_rate: 31250
                                dev.ix.0.queue1.tx_packets: 685675
                                dev.ix.0.queue1.tso_tx: 0
                                dev.ix.0.queue1.txd_tail: 1525
                                dev.ix.0.queue1.txd_head: 1525
                                dev.ix.0.queue0.rx_discarded: 0
                                dev.ix.0.queue0.rx_copies: 0
                                dev.ix.0.queue0.rx_bytes: 3775931277
                                dev.ix.0.queue0.rx_packets: 3162670
                                dev.ix.0.queue0.rxd_tail: 557
                                dev.ix.0.queue0.rxd_head: 558
                                dev.ix.0.queue0.irqs: 1053920
                                dev.ix.0.queue0.interrupt_rate: 31250
                                dev.ix.0.queue0.tx_packets: 901500
                                dev.ix.0.queue0.tso_tx: 0
                                dev.ix.0.queue0.txd_tail: 946
                                dev.ix.0.queue0.txd_head: 946
                                dev.ix.0.link_irq: 3
                                dev.ix.0.watchdog_events: 0
                                dev.ix.0.dropped: 0
                                dev.ix.0.iflib.rxq3.rxq_fl0.buf_size: 2048
                                dev.ix.0.iflib.rxq3.rxq_fl0.credits: 2047
                                dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 245
                                dev.ix.0.iflib.rxq3.rxq_fl0.pidx: 244
                                dev.ix.0.iflib.rxq3.cpu: 3
                                dev.ix.0.iflib.rxq2.rxq_fl0.buf_size: 2048
                                dev.ix.0.iflib.rxq2.rxq_fl0.credits: 128
                                dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 0
                                dev.ix.0.iflib.rxq2.rxq_fl0.pidx: 128
                                dev.ix.0.iflib.rxq2.cpu: 2
                                dev.ix.0.iflib.rxq1.rxq_fl0.buf_size: 2048
                                dev.ix.0.iflib.rxq1.rxq_fl0.credits: 128
                                dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 0
                                dev.ix.0.iflib.rxq1.rxq_fl0.pidx: 128
                                dev.ix.0.iflib.rxq1.cpu: 1
                                dev.ix.0.iflib.rxq0.rxq_fl0.buf_size: 2048
                                dev.ix.0.iflib.rxq0.rxq_fl0.credits: 2047
                                dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 558
                                dev.ix.0.iflib.rxq0.rxq_fl0.pidx: 557
                                dev.ix.0.iflib.rxq0.cpu: 0
                                dev.ix.0.iflib.txq3.r_abdications: 0
                                dev.ix.0.iflib.txq3.r_restarts: 0
                                dev.ix.0.iflib.txq3.r_stalls: 0
                                dev.ix.0.iflib.txq3.r_starts: 1027076
                                dev.ix.0.iflib.txq3.r_drops: 0
                                dev.ix.0.iflib.txq3.r_enqueues: 1027315
                                dev.ix.0.iflib.txq3.ring_state: pidx_head: 1267 pidx_tail: 1267 cidx: 1267 state: IDLE
                                dev.ix.0.iflib.txq3.txq_cleaned: 2211617
                                dev.ix.0.iflib.txq3.txq_processed: 2211649
                                dev.ix.0.iflib.txq3.txq_in_use: 34
                                dev.ix.0.iflib.txq3.txq_cidx_processed: 1857
                                dev.ix.0.iflib.txq3.txq_cidx: 1825
                                dev.ix.0.iflib.txq3.txq_pidx: 1859
                                dev.ix.0.iflib.txq3.no_tx_dma_setup: 0
                                dev.ix.0.iflib.txq3.txd_encap_efbig: 0
                                dev.ix.0.iflib.txq3.tx_map_failed: 0
                                dev.ix.0.iflib.txq3.no_desc_avail: 0
                                dev.ix.0.iflib.txq3.mbuf_defrag_failed: 0
                                dev.ix.0.iflib.txq3.m_pullups: 0
                                dev.ix.0.iflib.txq3.mbuf_defrag: 0
                                dev.ix.0.iflib.txq3.cpu: 3
                                dev.ix.0.iflib.txq2.r_abdications: 0
                                dev.ix.0.iflib.txq2.r_restarts: 0
                                dev.ix.0.iflib.txq2.r_stalls: 0
                                dev.ix.0.iflib.txq2.r_starts: 1189683
                                dev.ix.0.iflib.txq2.r_drops: 0
                                dev.ix.0.iflib.txq2.r_enqueues: 1190417
                                dev.ix.0.iflib.txq2.ring_state: pidx_head: 0529 pidx_tail: 0529 cidx: 0529 state: IDLE
                                dev.ix.0.iflib.txq2.txq_cleaned: 2858757
                                dev.ix.0.iflib.txq2.txq_processed: 2858789
                                dev.ix.0.iflib.txq2.txq_in_use: 34
                                dev.ix.0.iflib.txq2.txq_cidx_processed: 1829
                                dev.ix.0.iflib.txq2.txq_cidx: 1797
                                dev.ix.0.iflib.txq2.txq_pidx: 1831
                                dev.ix.0.iflib.txq2.no_tx_dma_setup: 0
                                dev.ix.0.iflib.txq2.txd_encap_efbig: 0
                                dev.ix.0.iflib.txq2.tx_map_failed: 0
                                dev.ix.0.iflib.txq2.no_desc_avail: 0
                                dev.ix.0.iflib.txq2.mbuf_defrag_failed: 0
                                dev.ix.0.iflib.txq2.m_pullups: 0
                                dev.ix.0.iflib.txq2.mbuf_defrag: 0
                                dev.ix.0.iflib.txq2.cpu: 2
                                dev.ix.0.iflib.txq1.r_abdications: 0
                                dev.ix.0.iflib.txq1.r_restarts: 0
                                dev.ix.0.iflib.txq1.r_stalls: 0
                                dev.ix.0.iflib.txq1.r_starts: 685820
                                dev.ix.0.iflib.txq1.r_drops: 0
                                dev.ix.0.iflib.txq1.r_enqueues: 686291
                                dev.ix.0.iflib.txq1.ring_state: pidx_head: 0211 pidx_tail: 0211 cidx: 0211 state: IDLE
                                dev.ix.0.iflib.txq1.txq_cleaned: 1818067
                                dev.ix.0.iflib.txq1.txq_processed: 1818099
                                dev.ix.0.iflib.txq1.txq_in_use: 34
                                dev.ix.0.iflib.txq1.txq_cidx_processed: 1523
                                dev.ix.0.iflib.txq1.txq_cidx: 1491
                                dev.ix.0.iflib.txq1.txq_pidx: 1525
                                dev.ix.0.iflib.txq1.no_tx_dma_setup: 0
                                dev.ix.0.iflib.txq1.txd_encap_efbig: 0
                                dev.ix.0.iflib.txq1.tx_map_failed: 0
                                dev.ix.0.iflib.txq1.no_desc_avail: 0
                                dev.ix.0.iflib.txq1.mbuf_defrag_failed: 0
                                dev.ix.0.iflib.txq1.m_pullups: 0
                                dev.ix.0.iflib.txq1.mbuf_defrag: 0
                                dev.ix.0.iflib.txq1.cpu: 1
                                dev.ix.0.iflib.txq0.r_abdications: 0
                                dev.ix.0.iflib.txq0.r_restarts: 0
                                dev.ix.0.iflib.txq0.r_stalls: 0
                                dev.ix.0.iflib.txq0.r_starts: 900768
                                dev.ix.0.iflib.txq0.r_drops: 0
                                dev.ix.0.iflib.txq0.r_enqueues: 901527
                                dev.ix.0.iflib.txq0.ring_state: pidx_head: 0407 pidx_tail: 0407 cidx: 0407 state: IDLE
                                dev.ix.0.iflib.txq0.txq_cleaned: 2419599
                                dev.ix.0.iflib.txq0.txq_processed: 2419631
                                dev.ix.0.iflib.txq0.txq_in_use: 35
                                dev.ix.0.iflib.txq0.txq_cidx_processed: 943
                                dev.ix.0.iflib.txq0.txq_cidx: 911
                                dev.ix.0.iflib.txq0.txq_pidx: 946
                                dev.ix.0.iflib.txq0.no_tx_dma_setup: 0
                                dev.ix.0.iflib.txq0.txd_encap_efbig: 0
                                dev.ix.0.iflib.txq0.tx_map_failed: 0
                                dev.ix.0.iflib.txq0.no_desc_avail: 0
                                dev.ix.0.iflib.txq0.mbuf_defrag_failed: 0
                                dev.ix.0.iflib.txq0.m_pullups: 0
                                dev.ix.0.iflib.txq0.mbuf_defrag: 0
                                dev.ix.0.iflib.txq0.cpu: 0
                                dev.ix.0.iflib.override_nrxds: 0
                                dev.ix.0.iflib.override_ntxds: 0
                                dev.ix.0.iflib.allocated_msix_vectors: 5
                                dev.ix.0.iflib.use_extra_msix_vectors: 0
                                dev.ix.0.iflib.use_logical_cores: 0
                                dev.ix.0.iflib.separate_txrx: 0
                                dev.ix.0.iflib.core_offset: 0
                                dev.ix.0.iflib.tx_abdicate: 0
                                dev.ix.0.iflib.rx_budget: 0
                                dev.ix.0.iflib.disable_msix: 0
                                dev.ix.0.iflib.override_qs_enable: 0
                                dev.ix.0.iflib.override_nrxqs: 0
                                dev.ix.0.iflib.override_ntxqs: 0
                                dev.ix.0.iflib.driver_version: 4.0.1-k
                                dev.ix.0.%iommu: rid=0x300
                                dev.ix.0.%parent: pci9
                                dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000 class=0x020000
                                dev.ix.0.%location: slot=0 function=0 dbsf=pci0:3:0:0 handle=\_SB_.PCI0.VRP0.LAN0
                                dev.ix.0.%driver: ix
                                dev.ix.0.%desc: Intel(R) X553 N (SFP+)
                                
                                1 Reply Last reply Reply Quote 0
                                • stephenw10S Offline
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  OK similar to the other user, no one error type:

                                  dev.ix.0.mac_stats.rx_missed_packets: 0
                                  dev.ix.0.mac_stats.rec_len_errs: 0
                                  dev.ix.0.mac_stats.remote_faults: 0
                                  dev.ix.0.mac_stats.local_faults: 1
                                  dev.ix.0.mac_stats.short_discards: 7
                                  dev.ix.0.mac_stats.byte_errs: 9220
                                  dev.ix.0.mac_stats.ill_errs: 1617
                                  dev.ix.0.mac_stats.crc_errs: 87158
                                  dev.ix.0.mac_stats.rx_errs: 90003
                                  dev.ix.0.watchdog_events: 0
                                  dev.ix.0.dropped: 0
                                  

                                  However what's different is that that isn't an XPON module it's just an 10GbaseT module.

                                  Technically neither type is supported by the X553 NICs because they don't have the data lines to detect the link type.

                                  What exactly is that connected to? At what speed?

                                  L 1 Reply Last reply Reply Quote 0
                                  • L Offline
                                    lio1503 @stephenw10
                                    last edited by

                                    @stephenw10

                                    It's connected to my ISP router on 10G Ethernet port.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S Offline
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Is the ISP router actually routing? Is it actually linked at 10G?

                                      One option here would be to just use one of the IGC ports. But that would be limited to 2.5Gbps of course.

                                      L 1 Reply Last reply Reply Quote 0
                                      • L Offline
                                        lio1503 @stephenw10
                                        last edited by

                                        @stephenw10

                                        No I just use ISP router for the PPPOE. Yes its linked to 10G.
                                        Finally I order a XSPON module to completely bypass the ISP router, while waiting for the module to arrive I use the IGC port and it's works good.
                                        Thanks you very much for your help.

                                        1 Reply Last reply Reply Quote 1
                                        • stephenw10S Offline
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Yup those RJ-45 SFP+ modules are not supported and actual function can be very variable. Some users have reported success with some modules but there's no real way to know any many don't work as expected.

                                          L 1 Reply Last reply Reply Quote 0
                                          • L Offline
                                            lio1503 @stephenw10
                                            last edited by

                                            @stephenw10
                                            OK.Thanks

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.