Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    8200/6100 Intel x553 NIC and certain SFP+ modules

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    25 Posts 3 Posters 940 Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      aholmes5
      last edited by

      I have a netgate 8200 which uses the intel atom c3000 SOC with built in intel x553 Ethernet for its SFP+ ports. When I use my XGS-PON module in either of the SFP+ ports, my download speed is very slow and deteriorates over minutes to maybe 5% of expected throughput. Upload speed always remains fast. By moving the PON module to a cheap unmanaged switch with two SFP+ ports and using a passive DAC cable to connect the switch to the 8200, throughput is normal. Based on what I've read this seems to be a driver issue in BSD with the x553. I'm guessing the 6100 is also affected since it also has an Atom C3000 series with the Intel x553.

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by

        What speeds do you usually get? Do you see any errors on the interface?

        Yes the 6100 would be identical in that respect.

        1 Reply Last reply Reply Quote 0
        • A Offline
          aholmes5
          last edited by

          I'm on a 2gbps symmetric fiber plan. I get 2300mbps up and down when I use my extra switch with a dac cable in the 8200's WAN port. When I plug the PON module into the WAN port directly (doesn't matter ix0 or ix1), I get upload speeds of 2300mbps but download speeds will be 1500mbps then within a minute or two deteriorate to 30-40mbps.

          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            No errors/collisions on the interface (Status > Interfaces) when that happens?

            A 1 Reply Last reply Reply Quote 0
            • A Offline
              aholmes5 @stephenw10
              last edited by

              @stephenw10 Lots of In errors. No out errors and no collisions.

              1 Reply Last reply Reply Quote 0
              • stephenw10S Offline
                stephenw10 Netgate Administrator
                last edited by

                Interesting. Then check the stats in the sysctl to see what sort of errors they are: sysctl dev.ix.0

                The C3K NICs are limited in the information they can get from modules about the link state. That can mean they negotiate incorrectly or fail to link with some media types. XPON modules that that hide the real link and present to the NIC as something else so there are a few things there that could cause problems.

                A 1 Reply Last reply Reply Quote 0
                • A Offline
                  aholmes5 @stephenw10
                  last edited by aholmes5

                  @stephenw10 SFP in 8200.txt SFP in switch.txt

                  Here is what I have. I don't know how to interpret this. Lots of errors:
                  dev.ix.0.mac_stats.rec_len_errs: 20
                  dev.ix.0.mac_stats.remote_faults: 0
                  dev.ix.0.mac_stats.local_faults: 53
                  dev.ix.0.mac_stats.short_discards: 7
                  dev.ix.0.mac_stats.byte_errs: 1425
                  dev.ix.0.mac_stats.ill_errs: 873
                  dev.ix.0.mac_stats.crc_errs: 5468
                  dev.ix.0.mac_stats.rx_errs: 6374

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    Mmm, interesting no one type....

                    Check netstat -m make sure there's no buffer exhaustion happening.

                    A 1 Reply Last reply Reply Quote 0
                    • A Offline
                      aholmes5 @stephenw10
                      last edited by

                      @stephenw10
                      58799/4456/63255 mbufs in use (current/cache/total)
                      49273/3559/52832/1000000 mbuf clusters in use (current/cache/total/max)
                      3/2029 mbuf+clusters out of packet secondary zone in use (current/cache)
                      0/1524/1524/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
                      0/0/0/524288 9k jumbo clusters in use (current/cache/total/max)
                      0/0/0/84161 16k jumbo clusters in use (current/cache/total/max)
                      113245K/14328K/127573K bytes allocated to network (current/cache/total)
                      0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
                      0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
                      0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
                      0/0/0 requests for jumbo clusters denied (4k/9k/16k)
                      0 sendfile syscalls
                      0 sendfile syscalls completed without I/O request
                      0 requests for I/O initiated by sendfile
                      0 pages read by sendfile as part of a request
                      0 pages were valid at time of a sendfile request
                      0 pages were valid and substituted to bogus page
                      0 pages were requested for read ahead by applications
                      0 pages were read ahead by sendfile
                      0 times sendfile encountered an already busy page
                      0 requests for sfbufs denied
                      0 requests for sfbufs delayed

                      1 Reply Last reply Reply Quote 0
                      • L Offline
                        lio1503
                        last edited by

                        Hi @aholmes5

                        I have the same issue. I connect my Fiber WAN on SFP+ 10G on netgate 6100 with a 10GTEK sfp+10G and my unifi switch on the second WAN on SFP+ 10G on netgate 6100 with DAC 10G cable but my download speed is reduce of 80% but my upload is fine.
                        Do you found a solution ?

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S Offline
                          stephenw10 Netgate Administrator
                          last edited by

                          Are you seeing errors on the interfaces?

                          Do you have any traffic shaping?

                          L 1 Reply Last reply Reply Quote 0
                          • L Offline
                            lio1503 @stephenw10
                            last edited by lio1503

                            @stephenw10
                            I see some error on interfaces and no traffic shaping configured.

                            1 Reply Last reply Reply Quote 0
                            • L Offline
                              lio1503
                              last edited by

                              Screenshot 2025-08-06 203441.png

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S Offline
                                stephenw10 Netgate Administrator
                                last edited by

                                No errors shown there. So some on the WAN?

                                How are you testing the throughput?

                                L 1 Reply Last reply Reply Quote 0
                                • L Offline
                                  lio1503 @stephenw10
                                  last edited by

                                  @stephenw10

                                  I put the wrong image sorry

                                  netgate1.png netgate2.png netgate3.png

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S Offline
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, not really a significant number there. Nothing that would cause throttling to that extent.

                                    What about the PPPoE parent interface? Which I assume is ix0?
                                    Try: netstat -i and ifconfig -v ix0

                                    L 1 Reply Last reply Reply Quote 0
                                    • L Offline
                                      lio1503 @stephenw10
                                      last edited by

                                      Hi @stephenw10

                                      Here the result of commands

                                      Screenshot 2025-08-08 112744.png

                                      Screenshot 2025-08-08 112955.png

                                      Thanks

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S Offline
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Ok huge number of input errors on ix0. That's what you'd expect for bad downloads.

                                        You have an XPON module in there I assume? Can you access that to get stats?

                                        You can check the sysctl output for ix0 to see what type of errors those are: sysctl dev.ix.0

                                        L 1 Reply Last reply Reply Quote 0
                                        • L Offline
                                          lio1503 @stephenw10
                                          last edited by stephenw10

                                          @stephenw10

                                          Yes I use this module on ix0 https://www.amazon.ca/dp/B06XG9DPJ7?th=1
                                          and this cable to unifi switch for Lan on ix1 https://www.amazon.ca/dp/B01M09C9NZ?th=1

                                          Result of sysctl dev.ix.0

                                          dev.ix.0.wake: 0
dev.ix.0.dmac: 0
dev.ix.0.tso_tcp_flags_mask_last_segment: 3967
dev.ix.0.tso_tcp_flags_mask_middle_segment: 4086
dev.ix.0.tso_tcp_flags_mask_first_segment: 4086
dev.ix.0.fw_version: nvm 0.58.0 eTrack 0x8000084b
dev.ix.0.enable_aim: 0
dev.ix.0.advertise_speed: 0
dev.ix.0.fc: 3
dev.ix.0.mac_stats.tx_frames_1024_1522: 2688329
dev.ix.0.mac_stats.tx_frames_512_1023: 9258
dev.ix.0.mac_stats.tx_frames_256_511: 7406
dev.ix.0.mac_stats.tx_frames_128_255: 51767
dev.ix.0.mac_stats.tx_frames_65_127: 1044399
dev.ix.0.mac_stats.tx_frames_64: 2651
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.mcast_pkts_txd: 5
dev.ix.0.mac_stats.bcast_pkts_txd: 1
dev.ix.0.mac_stats.good_pkts_txd: 3803810
dev.ix.0.mac_stats.total_pkts_txd: 3803810
dev.ix.0.mac_stats.good_octets_txd: 4177580394
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.recv_jabberd: 1228
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.rx_frames_1024_1522: 2492339
dev.ix.0.mac_stats.rx_frames_512_1023: 8772
dev.ix.0.mac_stats.rx_frames_256_511: 8173
dev.ix.0.mac_stats.rx_frames_128_255: 30310
dev.ix.0.mac_stats.rx_frames_65_127: 620807
dev.ix.0.mac_stats.rx_frames_64: 2487
dev.ix.0.mac_stats.bcast_pkts_rcvd: 247
dev.ix.0.mac_stats.mcast_pkts_rcvd: 10
dev.ix.0.mac_stats.good_pkts_rcvd: 3162888
dev.ix.0.mac_stats.total_pkts_rcvd: 3260844
dev.ix.0.mac_stats.good_octets_rcvd: 3788604566
dev.ix.0.mac_stats.total_octets_rcvd: 3930535548
dev.ix.0.mac_stats.xoff_recvd: 0
dev.ix.0.mac_stats.xoff_txd: 0
dev.ix.0.mac_stats.xon_recvd: 0
dev.ix.0.mac_stats.xon_txd: 0
dev.ix.0.mac_stats.rx_missed_packets: 0
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.remote_faults: 0
dev.ix.0.mac_stats.local_faults: 1
dev.ix.0.mac_stats.short_discards: 7
dev.ix.0.mac_stats.byte_errs: 9220
dev.ix.0.mac_stats.ill_errs: 1617
dev.ix.0.mac_stats.crc_errs: 87158
dev.ix.0.mac_stats.rx_errs: 90003
dev.ix.0.queue3.rx_discarded: 0
dev.ix.0.queue3.rx_copies: 0
dev.ix.0.queue3.rx_bytes: 25480
dev.ix.0.queue3.rx_packets: 245
dev.ix.0.queue3.rxd_tail: 244
dev.ix.0.queue3.rxd_head: 245
dev.ix.0.queue3.irqs: 482085
dev.ix.0.queue3.interrupt_rate: 31250
dev.ix.0.queue3.tx_packets: 1027008
dev.ix.0.queue3.tso_tx: 0
dev.ix.0.queue3.txd_tail: 1859
dev.ix.0.queue3.txd_head: 1859
dev.ix.0.queue2.rx_discarded: 0
dev.ix.0.queue2.rx_copies: 0
dev.ix.0.queue2.rx_bytes: 0
dev.ix.0.queue2.rx_packets: 0
dev.ix.0.queue2.rxd_tail: 128
dev.ix.0.queue2.rxd_head: 0
dev.ix.0.queue2.irqs: 594318
dev.ix.0.queue2.interrupt_rate: 31250
dev.ix.0.queue2.tx_packets: 1189658
dev.ix.0.queue2.tso_tx: 0
dev.ix.0.queue2.txd_tail: 1831
dev.ix.0.queue2.txd_head: 1831
dev.ix.0.queue1.rx_discarded: 0
dev.ix.0.queue1.rx_copies: 0
dev.ix.0.queue1.rx_bytes: 0
dev.ix.0.queue1.rx_packets: 0
dev.ix.0.queue1.rxd_tail: 128
dev.ix.0.queue1.rxd_head: 0
dev.ix.0.queue1.irqs: 414076
dev.ix.0.queue1.interrupt_rate: 31250
dev.ix.0.queue1.tx_packets: 685675
dev.ix.0.queue1.tso_tx: 0
dev.ix.0.queue1.txd_tail: 1525
dev.ix.0.queue1.txd_head: 1525
dev.ix.0.queue0.rx_discarded: 0
dev.ix.0.queue0.rx_copies: 0
dev.ix.0.queue0.rx_bytes: 3775931277
dev.ix.0.queue0.rx_packets: 3162670
dev.ix.0.queue0.rxd_tail: 557
dev.ix.0.queue0.rxd_head: 558
dev.ix.0.queue0.irqs: 1053920
dev.ix.0.queue0.interrupt_rate: 31250
dev.ix.0.queue0.tx_packets: 901500
dev.ix.0.queue0.tso_tx: 0
dev.ix.0.queue0.txd_tail: 946
dev.ix.0.queue0.txd_head: 946
dev.ix.0.link_irq: 3
dev.ix.0.watchdog_events: 0
dev.ix.0.dropped: 0
dev.ix.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq3.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 245
dev.ix.0.iflib.rxq3.rxq_fl0.pidx: 244
dev.ix.0.iflib.rxq3.cpu: 3
dev.ix.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq2.rxq_fl0.credits: 128
dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 0
dev.ix.0.iflib.rxq2.rxq_fl0.pidx: 128
dev.ix.0.iflib.rxq2.cpu: 2
dev.ix.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq1.rxq_fl0.credits: 128
dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.ix.0.iflib.rxq1.rxq_fl0.pidx: 128
dev.ix.0.iflib.rxq1.cpu: 1
dev.ix.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq0.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 558
dev.ix.0.iflib.rxq0.rxq_fl0.pidx: 557
dev.ix.0.iflib.rxq0.cpu: 0
dev.ix.0.iflib.txq3.r_abdications: 0
dev.ix.0.iflib.txq3.r_restarts: 0
dev.ix.0.iflib.txq3.r_stalls: 0
dev.ix.0.iflib.txq3.r_starts: 1027076
dev.ix.0.iflib.txq3.r_drops: 0
dev.ix.0.iflib.txq3.r_enqueues: 1027315
dev.ix.0.iflib.txq3.ring_state: pidx_head: 1267 pidx_tail: 1267 cidx: 1267 state: IDLE
dev.ix.0.iflib.txq3.txq_cleaned: 2211617
dev.ix.0.iflib.txq3.txq_processed: 2211649
dev.ix.0.iflib.txq3.txq_in_use: 34
dev.ix.0.iflib.txq3.txq_cidx_processed: 1857
dev.ix.0.iflib.txq3.txq_cidx: 1825
dev.ix.0.iflib.txq3.txq_pidx: 1859
dev.ix.0.iflib.txq3.no_tx_dma_setup: 0
dev.ix.0.iflib.txq3.txd_encap_efbig: 0
dev.ix.0.iflib.txq3.tx_map_failed: 0
dev.ix.0.iflib.txq3.no_desc_avail: 0
dev.ix.0.iflib.txq3.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq3.m_pullups: 0
dev.ix.0.iflib.txq3.mbuf_defrag: 0
dev.ix.0.iflib.txq3.cpu: 3
dev.ix.0.iflib.txq2.r_abdications: 0
dev.ix.0.iflib.txq2.r_restarts: 0
dev.ix.0.iflib.txq2.r_stalls: 0
dev.ix.0.iflib.txq2.r_starts: 1189683
dev.ix.0.iflib.txq2.r_drops: 0
dev.ix.0.iflib.txq2.r_enqueues: 1190417
dev.ix.0.iflib.txq2.ring_state: pidx_head: 0529 pidx_tail: 0529 cidx: 0529 state: IDLE
dev.ix.0.iflib.txq2.txq_cleaned: 2858757
dev.ix.0.iflib.txq2.txq_processed: 2858789
dev.ix.0.iflib.txq2.txq_in_use: 34
dev.ix.0.iflib.txq2.txq_cidx_processed: 1829
dev.ix.0.iflib.txq2.txq_cidx: 1797
dev.ix.0.iflib.txq2.txq_pidx: 1831
dev.ix.0.iflib.txq2.no_tx_dma_setup: 0
dev.ix.0.iflib.txq2.txd_encap_efbig: 0
dev.ix.0.iflib.txq2.tx_map_failed: 0
dev.ix.0.iflib.txq2.no_desc_avail: 0
dev.ix.0.iflib.txq2.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq2.m_pullups: 0
dev.ix.0.iflib.txq2.mbuf_defrag: 0
dev.ix.0.iflib.txq2.cpu: 2
dev.ix.0.iflib.txq1.r_abdications: 0
dev.ix.0.iflib.txq1.r_restarts: 0
dev.ix.0.iflib.txq1.r_stalls: 0
dev.ix.0.iflib.txq1.r_starts: 685820
dev.ix.0.iflib.txq1.r_drops: 0
dev.ix.0.iflib.txq1.r_enqueues: 686291
dev.ix.0.iflib.txq1.ring_state: pidx_head: 0211 pidx_tail: 0211 cidx: 0211 state: IDLE
dev.ix.0.iflib.txq1.txq_cleaned: 1818067
dev.ix.0.iflib.txq1.txq_processed: 1818099
dev.ix.0.iflib.txq1.txq_in_use: 34
dev.ix.0.iflib.txq1.txq_cidx_processed: 1523
dev.ix.0.iflib.txq1.txq_cidx: 1491
dev.ix.0.iflib.txq1.txq_pidx: 1525
dev.ix.0.iflib.txq1.no_tx_dma_setup: 0
dev.ix.0.iflib.txq1.txd_encap_efbig: 0
dev.ix.0.iflib.txq1.tx_map_failed: 0
dev.ix.0.iflib.txq1.no_desc_avail: 0
dev.ix.0.iflib.txq1.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq1.m_pullups: 0
dev.ix.0.iflib.txq1.mbuf_defrag: 0
dev.ix.0.iflib.txq1.cpu: 1
dev.ix.0.iflib.txq0.r_abdications: 0
dev.ix.0.iflib.txq0.r_restarts: 0
dev.ix.0.iflib.txq0.r_stalls: 0
dev.ix.0.iflib.txq0.r_starts: 900768
dev.ix.0.iflib.txq0.r_drops: 0
dev.ix.0.iflib.txq0.r_enqueues: 901527
dev.ix.0.iflib.txq0.ring_state: pidx_head: 0407 pidx_tail: 0407 cidx: 0407 state: IDLE
dev.ix.0.iflib.txq0.txq_cleaned: 2419599
dev.ix.0.iflib.txq0.txq_processed: 2419631
dev.ix.0.iflib.txq0.txq_in_use: 35
dev.ix.0.iflib.txq0.txq_cidx_processed: 943
dev.ix.0.iflib.txq0.txq_cidx: 911
dev.ix.0.iflib.txq0.txq_pidx: 946
dev.ix.0.iflib.txq0.no_tx_dma_setup: 0
dev.ix.0.iflib.txq0.txd_encap_efbig: 0
dev.ix.0.iflib.txq0.tx_map_failed: 0
dev.ix.0.iflib.txq0.no_desc_avail: 0
dev.ix.0.iflib.txq0.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq0.m_pullups: 0
dev.ix.0.iflib.txq0.mbuf_defrag: 0
dev.ix.0.iflib.txq0.cpu: 0
dev.ix.0.iflib.override_nrxds: 0
dev.ix.0.iflib.override_ntxds: 0
dev.ix.0.iflib.allocated_msix_vectors: 5
dev.ix.0.iflib.use_extra_msix_vectors: 0
dev.ix.0.iflib.use_logical_cores: 0
dev.ix.0.iflib.separate_txrx: 0
dev.ix.0.iflib.core_offset: 0
dev.ix.0.iflib.tx_abdicate: 0
dev.ix.0.iflib.rx_budget: 0
dev.ix.0.iflib.disable_msix: 0
dev.ix.0.iflib.override_qs_enable: 0
dev.ix.0.iflib.override_nrxqs: 0
dev.ix.0.iflib.override_ntxqs: 0
dev.ix.0.iflib.driver_version: 4.0.1-k
dev.ix.0.%iommu: rid=0x300
dev.ix.0.%parent: pci9
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:3:0:0 handle=\_SB_.PCI0.VRP0.LAN0
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) X553 N (SFP+)
                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S Offline
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            OK similar to the other user, no one error type:

                                            dev.ix.0.mac_stats.rx_missed_packets: 0
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.remote_faults: 0
dev.ix.0.mac_stats.local_faults: 1
dev.ix.0.mac_stats.short_discards: 7
dev.ix.0.mac_stats.byte_errs: 9220
dev.ix.0.mac_stats.ill_errs: 1617
dev.ix.0.mac_stats.crc_errs: 87158
dev.ix.0.mac_stats.rx_errs: 90003
dev.ix.0.watchdog_events: 0
dev.ix.0.dropped: 0

                                            However what's different is that that isn't an XPON module it's just an 10GbaseT module.

                                            Technically neither type is supported by the X553 NICs because they don't have the data lines to detect the link type.

                                            What exactly is that connected to? At what speed?

                                            L 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.