• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Packet error on physical interface

Scheduled Pinned Locked Moved Hardware
14 Posts 2 Posters 696 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • W
    william_vn
    last edited by Feb 16, 2023, 4:04 AM

    Hi,
    I am using Pfsense that was installed on Dell Server, and it's working on production.
    But I got a lot of packet error on almost of interface, like bellow:

    [2.6.0-RELEASE][admin@pfs2]/root: netstat -i

    Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
    
    bce2   1500 <Link#3>      b8:ac:6f:xx:xx:xx 949767111 889101     0 460586066     0     0
    bce2      - fe80::%bce2/6 fe80::baac:6fff:f        0     -     -        1     -     -
    bce3*  1500 <Link#4>      b8:ac:6f:97:06:f0        0     0     0        0     0     0
    ix0    1500 <Link#5>      00:1b:21:xx:xx:xx 4708069675 4415248     0 11655678303     0     0
    ix1    1500 <Link#6>      00:1b:21:xx:xx:xx 7296506329 12473470     0 11841483585     0     0
    
    

    for example: packet error rate for bce2 interface is: (889101/949767111)*100 = 0.9 %
    This problem happens frequently, sometimes to bce2 interface, sometimes to other interfaces.
    I am not sure this problem relate to hardware or just something wrong in setting of pfsense software.

    Anyone can help to give me some tips on troubleshooting for this problem ?
    Thanks in advance

    William

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by Feb 16, 2023, 2:47 PM

      The fact it's happening on all interfaces seems suspicious. The Intel ix NICs can report increased errors like that due to driver changes but I'm not aware of anything similar in bce.
      Are they all linked to the same thing?

      You can look at the mac stats in the sysctl output to see what type of errors those are, at least for ix. For example:

      dev.ix.0.mac_stats.recv_jabberd: 0
      dev.ix.0.mac_stats.recv_oversized: 0
      dev.ix.0.mac_stats.recv_fragmented: 0
      dev.ix.0.mac_stats.recv_undersized: 0
      dev.ix.0.mac_stats.rx_missed_packets: 0
      dev.ix.0.mac_stats.rec_len_errs: 0
      dev.ix.0.mac_stats.remote_faults: 0
      dev.ix.0.mac_stats.local_faults: 1
      dev.ix.0.mac_stats.short_discards: 0
      
      W 1 Reply Last reply Feb 21, 2023, 3:11 AM Reply Quote 0
      • W
        william_vn @stephenw10
        last edited by william_vn Feb 21, 2023, 3:19 AM Feb 21, 2023, 3:11 AM

        @stephenw10 hi,
        ix0 and ix1 are internal interfaces (LAN), and they were bonded into lagg0 interface.
        bce* and bge* are inferfaces used for internet (WAN interface). they are configured PPPOE.


        [2.6.0-RELEASE][admin@pfs2]/root: sysctl -a | grep mac_stats
        dev.ix.1.mac_stats.tx_frames_1024_1522: 11199336056
        dev.ix.1.mac_stats.tx_frames_512_1023: 296554491
        dev.ix.1.mac_stats.tx_frames_256_511: 267149362
        dev.ix.1.mac_stats.tx_frames_128_255: 469563788
        dev.ix.1.mac_stats.tx_frames_65_127: 1743645376
        dev.ix.1.mac_stats.tx_frames_64: 200622859
        dev.ix.1.mac_stats.management_pkts_txd: 0
        dev.ix.1.mac_stats.mcast_pkts_txd: 113559
        dev.ix.1.mac_stats.bcast_pkts_txd: 9892940
        dev.ix.1.mac_stats.good_pkts_txd: 14176871932
        dev.ix.1.mac_stats.total_pkts_txd: 14176872009
        dev.ix.1.mac_stats.good_octets_txd: 16582022206821
        dev.ix.1.mac_stats.checksum_errs: 16885390
        dev.ix.1.mac_stats.management_pkts_drpd: 0
        dev.ix.1.mac_stats.management_pkts_rcvd: 0
        dev.ix.1.mac_stats.recv_jabberd: 0
        dev.ix.1.mac_stats.recv_oversized: 0
        dev.ix.1.mac_stats.recv_fragmented: 0
        dev.ix.1.mac_stats.recv_undersized: 0
        dev.ix.1.mac_stats.rx_frames_1024_1522: 1627134501
        dev.ix.1.mac_stats.rx_frames_512_1023: 256050662
        dev.ix.1.mac_stats.rx_frames_256_511: 201144253
        dev.ix.1.mac_stats.rx_frames_128_255: 809553201
        dev.ix.1.mac_stats.rx_frames_65_127: 5598262630
        dev.ix.1.mac_stats.rx_frames_64: 218715925
        dev.ix.1.mac_stats.bcast_pkts_rcvd: 120420301
        dev.ix.1.mac_stats.mcast_pkts_rcvd: 111525769
        dev.ix.1.mac_stats.good_pkts_rcvd: 8710861172
        dev.ix.1.mac_stats.total_pkts_rcvd: 9968563894
        dev.ix.1.mac_stats.good_octets_rcvd: 3170640992241
        dev.ix.1.mac_stats.total_octets_rcvd: 3516919350290
        dev.ix.1.mac_stats.xoff_recvd: 0
        dev.ix.1.mac_stats.xoff_txd: 76
        dev.ix.1.mac_stats.xon_recvd: 0
        dev.ix.1.mac_stats.xon_txd: 1
        dev.ix.1.mac_stats.rx_missed_packets: 2371
        dev.ix.1.mac_stats.rec_len_errs: 0
        dev.ix.1.mac_stats.remote_faults: 4
        dev.ix.1.mac_stats.local_faults: 9775
        dev.ix.1.mac_stats.short_discards: 0
        dev.ix.1.mac_stats.byte_errs: 0
        dev.ix.1.mac_stats.ill_errs: 0
        dev.ix.1.mac_stats.crc_errs: 0
        dev.ix.1.mac_stats.rx_errs: 16887761
        dev.ix.0.mac_stats.tx_frames_1024_1522: 11117151076
        dev.ix.0.mac_stats.tx_frames_512_1023: 282499408
        dev.ix.0.mac_stats.tx_frames_256_511: 260685094
        dev.ix.0.mac_stats.tx_frames_128_255: 462588596
        dev.ix.0.mac_stats.tx_frames_65_127: 1733360311
        dev.ix.0.mac_stats.tx_frames_64: 155798861
        dev.ix.0.mac_stats.management_pkts_txd: 0
        dev.ix.0.mac_stats.mcast_pkts_txd: 113549
        dev.ix.0.mac_stats.bcast_pkts_txd: 11355561
        dev.ix.0.mac_stats.good_pkts_txd: 14012083346
        dev.ix.0.mac_stats.total_pkts_txd: 14012083485
        dev.ix.0.mac_stats.good_octets_txd: 16449667332507
        dev.ix.0.mac_stats.checksum_errs: 6506947
        dev.ix.0.mac_stats.management_pkts_drpd: 0
        dev.ix.0.mac_stats.management_pkts_rcvd: 0
        dev.ix.0.mac_stats.recv_jabberd: 0
        dev.ix.0.mac_stats.recv_oversized: 0
        dev.ix.0.mac_stats.recv_fragmented: 0
        dev.ix.0.mac_stats.recv_undersized: 0
        dev.ix.0.mac_stats.rx_frames_1024_1522: 1193629067
        dev.ix.0.mac_stats.rx_frames_512_1023: 176745206
        dev.ix.0.mac_stats.rx_frames_256_511: 145828688
        dev.ix.0.mac_stats.rx_frames_128_255: 396491922
        dev.ix.0.mac_stats.rx_frames_65_127: 3527593358
        dev.ix.0.mac_stats.rx_frames_64: 196177107
        dev.ix.0.mac_stats.bcast_pkts_rcvd: 987533
        dev.ix.0.mac_stats.mcast_pkts_rcvd: 1256931
        dev.ix.0.mac_stats.good_pkts_rcvd: 5636465348
        dev.ix.0.mac_stats.total_pkts_rcvd: 5754406086
        dev.ix.0.mac_stats.good_octets_rcvd: 2255966211057
        dev.ix.0.mac_stats.total_octets_rcvd: 2306898407455
        dev.ix.0.mac_stats.xoff_recvd: 0
        dev.ix.0.mac_stats.xoff_txd: 137
        dev.ix.0.mac_stats.xon_recvd: 0
        dev.ix.0.mac_stats.xon_txd: 2
        dev.ix.0.mac_stats.rx_missed_packets: 2312
        dev.ix.0.mac_stats.rec_len_errs: 0
        dev.ix.0.mac_stats.remote_faults: 3
        dev.ix.0.mac_stats.local_faults: 11548
        dev.ix.0.mac_stats.short_discards: 0
        dev.ix.0.mac_stats.byte_errs: 0
        dev.ix.0.mac_stats.ill_errs: 0
        dev.ix.0.mac_stats.crc_errs: 0
        dev.ix.0.mac_stats.rx_errs: 6509259
        
        

        bce2 Interface:

        [2.6.0-RELEASE][admin@pfs2]/root: sysctl -a | grep bce.2
        dev.bce.2.com_no_buffers: 1723435
        dev.bce.2.stat_CatchupInRuleCheckerP4Hit: 0
        dev.bce.2.stat_CatchupInMBUFDiscards: 0
        dev.bce.2.stat_CatchupInFTQDiscards: 0
        dev.bce.2.stat_CatchupInRuleCheckerDiscards: 0
        dev.bce.2.stat_IfInRuleCheckerP4Hit: 0
        dev.bce.2.stat_IfInMBUFDiscards: 0
        dev.bce.2.stat_IfInFTQDiscards: 0
        dev.bce.2.stat_IfInRuleCheckerDiscards: 0
        dev.bce.2.stat_IfInFramesL2FilterDiscards: 0
        dev.bce.2.stat_XoffStateEntered: 0
        dev.bce.2.stat_MacControlFramesReceived: 0
        dev.bce.2.stat_FlowControlDone: 0
        dev.bce.2.stat_OutXoffSent: 0
        dev.bce.2.stat_OutXonSent: 0
        dev.bce.2.stat_XoffPauseFramesReceived: 0
        dev.bce.2.stat_XonPauseFramesReceived: 0
        dev.bce.2.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
        dev.bce.2.stat_EtherStatsPktsTx1024Octetsto1522Octets: 94602036
        dev.bce.2.stat_EtherStatsPktsTx512Octetsto1023Octets: 20576003
        dev.bce.2.stat_EtherStatsPktsTx256Octetsto511Octets: 12982897
        dev.bce.2.stat_EtherStatsPktsTx128Octetsto255Octets: 64841148
        dev.bce.2.stat_EtherStatsPktsTx65Octetsto127Octets: 486941526
        dev.bce.2.stat_EtherStatsPktsTx64Octets: 6797799
        dev.bce.2.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
        dev.bce.2.stat_EtherStatsPktsRx1024Octetsto1522Octets: 1157838817
        dev.bce.2.stat_EtherStatsPktsRx512Octetsto1023Octets: 27269494
        dev.bce.2.stat_EtherStatsPktsRx256Octetsto511Octets: 24201015
        dev.bce.2.stat_EtherStatsPktsRx128Octetsto255Octets: 42154606
        dev.bce.2.stat_EtherStatsPktsRx65Octetsto127Octets: 152389240
        dev.bce.2.stat_EtherStatsPktsRx64Octets: 7241567
        dev.bce.2.stat_EtherStatsOversizePkts: 0
        dev.bce.2.stat_EtherStatsUndersizePkts: 0
        dev.bce.2.stat_EtherStatsJabbers: 0
        dev.bce.2.stat_EtherStatsFragments: 0
        dev.bce.2.stat_EtherStatsCollisions: 0
        dev.bce.2.stat_Dot3StatsLateCollisions: 0
        dev.bce.2.stat_Dot3StatsExcessiveCollisions: 0
        dev.bce.2.stat_Dot3StatsDeferredTransmissions: 0
        dev.bce.2.stat_Dot3StatsMultipleCollisionFrames: 0
        dev.bce.2.stat_Dot3StatsSingleCollisionFrames: 0
        dev.bce.2.stat_Dot3StatsAlignmentErrors: 0
        dev.bce.2.stat_Dot3StatsFCSErrors: 0
        dev.bce.2.stat_Dot3StatsCarrierSenseErrors: 0
        dev.bce.2.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
        dev.bce.2.stat_IfHCOutBroadcastPkts: 3
        dev.bce.2.stat_IfHCOutMulticastPkts: 3
        dev.bce.2.stat_IfHCOutUcastPkts: 686741403
        dev.bce.2.stat_IfHCInBroadcastPkts: 0
        dev.bce.2.stat_IfHCInMulticastPkts: 0
        dev.bce.2.stat_IfHCInUcastPkts: 1411094739
        dev.bce.2.stat_IfHCOutBadOctets: 0
        dev.bce.2.stat_IfHCOutOctets: 203247572645
        dev.bce.2.stat_IfHCInBadOctets: 0
        dev.bce.2.stat_IfHcInOctets: 1701496006956
        dev.bce.2.unexpected_attention_count: 0
        dev.bce.2.dma_map_addr_tx_failed_count: 0
        dev.bce.2.dma_map_addr_rx_failed_count: 0
        dev.bce.2.mbuf_frag_count: 0
        dev.bce.2.mbuf_alloc_failed_count: 0
        dev.bce.2.l2fhdr_error_count: 0
        dev.bce.2.%parent: pci2
        dev.bce.2.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0235 class=0x020000
        dev.bce.2.%location: slot=0 function=0 dbsf=pci0:2:0:0
        dev.bce.2.%driver: bce
        dev.bce.2.%desc: QLogic NetXtreme II BCM5709 1000Base-T (C0)
        
        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator
          last edited by Feb 21, 2023, 2:36 PM

          Hmm, well looks like they are all checksum errors on the ix NICs. However that was updated int he driver for 2.6 and then further changes were made more recently as some of those were shown to be spurious.

          Looks it's buffer exhaustion on the bce NIC which normally means the NIC is not being serviced by the CPU fast enough for the incoming traffic.
          I would check that value matched the current input error count from netstat though.

          Steve

          W 1 Reply Last reply Feb 22, 2023, 4:33 AM Reply Quote 0
          • W
            william_vn @stephenw10
            last edited by Feb 22, 2023, 4:33 AM

            @stephenw10
            here is full output from sysctl and netstat:
            output.txt

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by Feb 22, 2023, 3:35 PM

              Hmm, OK so those errors are shown as InputDiscards for bge, as com_no_buffers for bce and as checksum_errs in ix.

              Do you have flow control active on any of those links?
              My first though here is that the NICs are just dropping packets because the CPU queues are maxed out.

              Steve

              W 1 Reply Last reply Feb 23, 2023, 3:11 AM Reply Quote 0
              • W
                william_vn @stephenw10
                last edited by Feb 23, 2023, 3:11 AM

                @stephenw10
                Do you have flow control active on any of those links?
                --> No, I don't

                My first though here is that the NICs are just dropping packets because the CPU queues are maxed out.
                --> so how can I check CPU queues ? and Is there solution for this?

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by Feb 23, 2023, 1:24 PM

                  Check the boot logs to make sure each NIC is coming up with the expected number of queues initially. Check the CPU usage per core on the firewall using top -HaSP at the CLI. Do you have cores that are maxed out.
                  Try enabling flow control. That should prevent packet loss in the even the NIC cannot be serviced fast enough. Though it may introduce other issues.

                  W 1 Reply Last reply Feb 24, 2023, 2:05 AM Reply Quote 0
                  • W
                    william_vn @stephenw10
                    last edited by Feb 24, 2023, 2:05 AM

                    @stephenw10 ,
                    top -HaSP

                    last pid: 81614;  load averages:  0.49,  0.57,  0.52                                                                                                  up 42+17:32:16  08:22:31
                    453 threads:   18 running, 384 sleeping, 51 waiting
                    CPU 0:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 1:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 2:   0.0% user,  0.0% nice, 16.7% system,  0.0% interrupt, 83.3% idle
                    CPU 3:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 4:   0.0% user,  0.0% nice, 16.7% system,  0.0% interrupt, 83.3% idle
                    CPU 5:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 6:   0.0% user,  0.0% nice,  8.3% system,  0.0% interrupt, 91.7% idle
                    CPU 7:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 8:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 9:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 10:  0.0% user,  0.0% nice,  7.7% system,  0.0% interrupt, 92.3% idle
                    CPU 11:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 12:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 13:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 14:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    CPU 15:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                    Mem: 106M Active, 6357M Inact, 1549M Wired, 763M Buf, 23G Free
                    Swap: 4096M Total, 4096M Free
                    
                      PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
                       11 root        155 ki31     0B   256K CPU9     9 1022.9 100.00% [idle{idle: cpu9}]
                       11 root        155 ki31     0B   256K CPU11   11 1022.8 100.00% [idle{idle: cpu11}]
                       11 root        155 ki31     0B   256K CPU15   15 1022.6 100.00% [idle{idle: cpu15}]
                       11 root        155 ki31     0B   256K CPU7     7 1022.4 100.00% [idle{idle: cpu7}]
                       11 root        155 ki31     0B   256K CPU1     1 1021.9 100.00% [idle{idle: cpu1}]
                       11 root        155 ki31     0B   256K CPU8     8 1016.7  96.53% [idle{idle: cpu8}]
                       11 root        155 ki31     0B   256K CPU10   10 1016.9  96.35% [idle{idle: cpu10}]
                       11 root        155 ki31     0B   256K CPU12   12 1016.5  96.32% [idle{idle: cpu12}]
                       11 root        155 ki31     0B   256K CPU14   14 1016.7  95.39% [idle{idle: cpu14}]
                       11 root        155 ki31     0B   256K CPU0     0 1000.3  94.90% [idle{idle: cpu0}]
                       11 root        155 ki31     0B   256K CPU3     3 1021.4  91.30% [idle{idle: cpu3}]
                       11 root        155 ki31     0B   256K CPU4     4 1004.4  91.23% [idle{idle: cpu4}]
                       11 root        155 ki31     0B   256K RUN      6 958.7H  88.47% [idle{idle: cpu6}]
                       11 root        155 ki31     0B   256K CPU2     2 1010.2  85.78% [idle{idle: cpu2}]
                       11 root        155 ki31     0B   256K CPU13   13 1022.7  85.24% [idle{idle: cpu13}]
                       11 root        155 ki31     0B   256K CPU5     5 1021.5  63.86% [idle{idle: cpu5}]
                        0 root        -92    -     0B  1616K -        2 420:57   8.32% [kernel{bge2 taskq}]
                        0 root        -92    -     0B  1616K -        1  61.3H   8.04% [kernel{bge0 taskq}]
                        0 root        -76    -     0B  1616K -        2 381:58   6.41% [kernel{if_io_tqg_2}]
                        0 root        -92    -     0B  1616K -        4 357:50   4.62% [kernel{bge3 taskq}]
                        0 root        -76    -     0B  1616K -       12 450:05   3.25% [kernel{if_io_tqg_12}]
                        0 root        -76    -     0B  1616K -        0 541:11   2.73% [kernel{if_io_tqg_0}]
                        0 root        -92    -     0B  1616K CPU0     0 906:30   2.41% [kernel{bge1 taskq}]
                       12 root        -92    -     0B   816K WAIT     4 502:59   2.20% [intr{irq258: bce2}]
                        0 root        -76    -     0B  1616K -       10 426:32   2.18% [kernel{if_io_tqg_10}]
                        0 root        -76    -     0B  1616K -        4 384:42   2.01% [kernel{if_io_tqg_4}]
                        0 root        -76    -     0B  1616K -        6 374:11   1.83% [kernel{if_io_tqg_6}]
                    81614 root         20    0    14M  4512K CPU6     6   0:00   1.74% top -HaSP
                        0 root        -76    -     0B  1616K -       14 430:22   1.57% [kernel{if_io_tqg_14}]
                        0 root        -76    -     0B  1616K -        8 440:59   1.49% [kernel{if_io_tqg_8}]
                    
                    
                    

                    cat dmesg.boot

                    ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xecc0-0xecdf mem 0xdf300000-0xdf37ffff,0xdf2f8000-0xdf2fbfff irq 35 at device 0.0 on pci5
                    ix0: Using 2048 TX descriptors and 2048 RX descriptors
                    ix0: Using 8 RX queues 8 TX queues
                    ix0: Using MSI-X interrupts with 9 vectors
                    ix0: allocated for 8 queues
                    ix0: allocated for 8 rx queues
                    ix0: Ethernet address: 00:1b:21:bc:77:c0
                    ix0: PCI Express Bus: Speed 5.0GT/s Width x4
                    ix0: eTrack 0x00012b2c PHY FW V65535
                    ix0: netmap queues/slots: TX 8/2048, RX 8/2048
                    ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xece0-0xecff mem 0xdf380000-0xdf3fffff,0xdf2fc000-0xdf2fffff irq 46 at device 0.1 on pci5
                    ix1: Using 2048 TX descriptors and 2048 RX descriptors
                    ix1: Using 8 RX queues 8 TX queues
                    ix1: Using MSI-X interrupts with 9 vectors
                    ix1: allocated for 8 queues
                    ix1: allocated for 8 rx queues
                    ix1: Ethernet address: 00:1b:21:bc:77:c1
                    ix1: PCI Express Bus: Speed 5.0GT/s Width x4
                    ix1: eTrack 0x00012b2c PHY FW V65535
                    ix1: netmap queues/slots: TX 8/2048, RX 8/2048
                    pcib6: <ACPI PCI-PCI bridge> at device 7.0 on pci0
                    pci6: <ACPI PCI bus> on pcib6
                    pcib7: <ACPI PCI-PCI bridge> at device 9.0 on pci0
                    pci7: <ACPI PCI bus> on pcib7
                    
                    

                    I find the ix0 and ix1 interface using 8 queues.

                    Additionally, I've re-checked and found pfsense already enabled flow control:

                    sysctl hw.ix

                    hw.ix.enable_rss: 1
                    hw.ix.enable_fdir: 0
                    hw.ix.unsupported_sfp: 0
                    hw.ix.enable_msix: 1
                    hw.ix.advertise_speed: 0
                    hw.ix.flow_control: 3
                    hw.ix.max_interrupt_rate: 31250
                    

                    Thanks!!!

                    1 Reply Last reply Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator
                      last edited by Feb 24, 2023, 1:12 PM

                      Flow control is negotiated though, both ends have to support it.

                      Nothing much happening in that top output. Was there any traffic flowing at that point? Were the errors increasing ?
                      What I expect to see is that when traffic through the NIC peaks the CPU core(s) servicing it max out and it drops packets. Assuming that is the cause.

                      Are you able to see the number of queues on the bce/bge NICs? vmstat -i may show it.

                      W 1 Reply Last reply Feb 27, 2023, 12:55 AM Reply Quote 0
                      • W
                        william_vn @stephenw10
                        last edited by Feb 27, 2023, 12:55 AM

                        @stephenw10
                        yes, a lot of traffic at that point, and the errors increasing as well.
                        CPU usage is always less than 10%.

                        Screen Shot 2023-02-27 at 07.52.14.png

                        [2.6.0-RELEASE][admin@pfs2]/root: vmstat -i
                        interrupt                          total       rate
                        irq17: uhci0                          24          0
                        irq19: ehci0                     2947641          1
                        irq23: atapci0                   2616920          1
                        cpu0:timer                    1228218976        311
                        cpu1:timer                      43088269         11
                        cpu2:timer                     515777307        131
                        cpu3:timer                      44479430         11
                        cpu4:timer                     665350682        168
                        cpu5:timer                      43841493         11
                        cpu6:timer                    1046412756        265
                        cpu7:timer                      41014489         10
                        cpu8:timer                     163311089         41
                        cpu9:timer                      42267453         11
                        cpu10:timer                    143573306         36
                        cpu11:timer                     43289875         11
                        cpu12:timer                    162167544         41
                        cpu13:timer                     44236981         11
                        cpu14:timer                    152160861         39
                        cpu15:timer                     45152052         11
                        irq257: bce1                     6913402          2
                        irq258: bce2                  2158281404        547
                        irq259: bce3                           1          0
                        irq260: mfi0                     7352305          2
                        irq261: ix0:rxq0              7415677330       1878
                        irq262: ix0:rxq1               709426817        180
                        irq263: ix0:rxq2               756092990        191
                        irq264: ix0:rxq3               701942814        178
                        irq265: ix0:rxq4               771849533        195
                        irq266: ix0:rxq5               709479026        180
                        irq267: ix0:rxq6               813804607        206
                        irq268: ix0:rxq7               701544714        178
                        irq269: ix0:aq                         6          0
                        irq270: ix1:rxq0              7808112326       1977
                        irq271: ix1:rxq1              1041604395        264
                        irq272: ix1:rxq2              1028264841        260
                        irq273: ix1:rxq3              1045890761        265
                        irq274: ix1:rxq4              1094897663        277
                        irq275: ix1:rxq5              1009812406        256
                        irq276: ix1:rxq6              1165305943        295
                        irq277: ix1:rxq7              1047942990        265
                        irq278: ix1:aq                         7          0
                        irq279: bge0                  6303565889       1596
                        irq280: bge1                  3249572863        823
                        irq281: bge2                  1628365893        412
                        irq282: bge3                  1149357028        291
                        Total                        46754965102      11839
                        [2.6.0-RELEASE][admin@pfs2]/root: netstat -i
                        Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
                        bce0*  1500 <Link#1>      b8:ac:6f:97:06:ea        0     0     0        0     0     0
                        bce1   1500 <Link#2>      b8:ac:6f:97:06:ec  7117685     0     0    70405     0     0
                        bce1      - fe80::%bce1/6 fe80::baac:6fff:f        0     -     -        1     -     -
                        bce1      - 192.168.103.0 pfs2               2343399     -     -      515     -     -
                        bce2   1500 <Link#3>      b8:ac:6f:97:06:ee 1868223693 2155425     0 906688134     0     0
                        bce2      - fe80::%bce2/6 fe80::baac:6fff:f        0     -     -        1     -     -
                        bce3   1500 <Link#4>      b8:ac:6f:97:06:f0        0     0     0        0     0     0
                        bce3      - fe80::%bce3/6 fe80::baac:6fff:f        0     -     -        1     -     -
                        bce3      - xxx.xxx.xx.xx xxx.xxx.xx.xx           0     -     -        0     -     -
                        ix0    1500 <Link#5>      00:1b:21:bc:77:c0 6590339737 7248161     0 16393394382     0     0
                        ix1    1500 <Link#6>      00:1b:21:bc:77:c0 10209454971 22546676     0 16548545943     0     0
                        bge0   1500 <Link#7>      00:0a:f7:90:e6:ec 21040184018 25001666     0 8758689519     0     0
                        bge0      - fe80::%bge0/6 fe80::20a:f7ff:fe        0     -     -        1     -     -
                        bge1   1500 <Link#8>      00:0a:f7:90:e6:ed 3801913176 6100037     0 2248351113     0     0
                        bge1      - fe80::%bge1/6 fe80::20a:f7ff:fe        0     -     -        0     -     -
                        bge2   1500 <Link#9>      00:0a:f7:90:e6:ee 1722390896 1717684     0 825525005     0     0
                        bge2      - fe80::%bge2/6 fe80::20a:f7ff:fe        0     -     -        1     -     -
                        bge3   1500 <Link#10>     00:0a:f7:90:e6:ef 1665701913 2291555     0 914404232     0     0
                        bge3      - fe80::%bge3/6 fe80::20a:f7ff:fe        0     -     -        2     -     -
                        
                        
                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by Feb 27, 2023, 1:22 AM

                          Hmm, looks like 1 irq per NIC for bce/bge. However I'd expect to see one CPU core at 100% if it was failing to service the NIC fast enough.

                          What are those NICs connected to? Are you seeing errors on the connected devices?

                          W 1 Reply Last reply Feb 27, 2023, 1:33 AM Reply Quote 0
                          • W
                            william_vn @stephenw10
                            last edited by Feb 27, 2023, 1:33 AM

                            @stephenw10 ,
                            here is output of top:

                            [2.6.0-RELEASE][admin@pfs2]/root: top -HaSP
                            
                            last pid: 11585;  load averages:  0.58,  0.76,  0.67                                                                                                    up 45+17:40:30  08:30:45
                            450 threads:   17 running, 382 sleeping, 51 waiting
                            CPU 0:   0.0% user,  0.0% nice, 12.8% system,  0.0% interrupt, 87.2% idle
                            CPU 1:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                            CPU 2:   0.0% user,  0.0% nice,  4.3% system,  0.0% interrupt, 95.7% idle
                            CPU 3:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                            CPU 4:   0.0% user,  0.0% nice,  8.0% system,  3.7% interrupt, 88.2% idle
                            CPU 5:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                            CPU 6:   0.0% user,  0.0% nice, 10.7% system,  0.0% interrupt, 89.3% idle
                            CPU 7:   0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                            CPU 8:   0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
                            CPU 9:   0.5% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.5% idle
                            CPU 10:  0.5% user,  0.0% nice,  2.7% system,  0.0% interrupt, 96.8% idle
                            CPU 11:  0.0% user,  0.0% nice,  0.5% system,  0.0% interrupt, 99.5% idle
                            CPU 12:  0.0% user,  0.0% nice,  2.1% system,  0.0% interrupt, 97.9% idle
                            CPU 13:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
                            CPU 14:  0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
                            CPU 15:  0.5% user,  0.0% nice,  0.5% system,  0.0% interrupt, 98.9% idle
                            Mem: 104M Active, 6370M Inact, 1570M Wired, 782M Buf, 23G Free
                            Swap: 4096M Total, 4096M Free
                            
                              PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
                               11 root        155 ki31     0B   256K CPU11   11 1094.8 100.00% [idle{idle: cpu11}]
                               11 root        155 ki31     0B   256K CPU13   13 1094.7 100.00% [idle{idle: cpu13}]
                               11 root        155 ki31     0B   256K CPU3     3 1093.3  99.84% [idle{idle: cpu3}]
                               11 root        155 ki31     0B   256K CPU1     1 1093.8  99.77% [idle{idle: cpu1}]
                               11 root        155 ki31     0B   256K CPU15   15 1094.6  99.46% [idle{idle: cpu15}]
                               11 root        155 ki31     0B   256K CPU7     7 1094.3  98.81% [idle{idle: cpu7}]
                               11 root        155 ki31     0B   256K CPU5     5 1093.5  98.73% [idle{idle: cpu5}]
                               11 root        155 ki31     0B   256K RUN      8 1088.5  98.21% [idle{idle: cpu8}]
                               11 root        155 ki31     0B   256K CPU14   14 1088.5  98.10% [idle{idle: cpu14}]
                               11 root        155 ki31     0B   256K CPU10   10 1088.6  97.99% [idle{idle: cpu10}]
                               11 root        155 ki31     0B   256K CPU12   12 1088.2  97.60% [idle{idle: cpu12}]
                               11 root        155 ki31     0B   256K CPU9     9 1094.8  97.09% [idle{idle: cpu9}]
                               11 root        155 ki31     0B   256K CPU2     2 1081.5  94.77% [idle{idle: cpu2}]
                               11 root        155 ki31     0B   256K CPU4     4 1075.0  92.37% [idle{idle: cpu4}]
                               11 root        155 ki31     0B   256K CPU0     0 1071.4  88.04% [idle{idle: cpu0}]
                               11 root        155 ki31     0B   256K CPU6     6 1028.6  85.34% [idle{idle: cpu6}]
                                0 root        -92    -     0B  1616K -        6  63.1H  12.78% [kernel{bge0 taskq}]
                                0 root        -92    -     0B  1616K -        0 938:32   8.50% [kernel{bge1 taskq}]
                                0 root        -76    -     0B  1616K -        0 564:06   3.74% [kernel{if_io_tqg_0}]
                               12 root        -92    -     0B   816K WAIT     4 533:51   3.09% [intr{irq258: bce2}]
                                0 root        -92    -     0B  1616K -        2 446:02   2.95% [kernel{bge2 taskq}]
                                0 root        -92    -     0B  1616K -        4 395:52   2.95% [kernel{bge3 taskq}]
                                0 root        -76    -     0B  1616K -       12 467:41   2.23% [kernel{if_io_tqg_12}]
                                0 root        -76    -     0B  1616K -        6 389:13   2.03% [kernel{if_io_tqg_6}]
                                0 root        -76    -     0B  1616K -        2 398:09   2.03% [kernel{if_io_tqg_2}]
                                0 root        -76    -     0B  1616K -       10 442:43   1.90% [kernel{if_io_tqg_10}]
                                0 root        -76    -     0B  1616K -       14 447:08   1.84% [kernel{if_io_tqg_14}]
                                0 root        -76    -     0B  1616K -        8 457:22   1.69% [kernel{if_io_tqg_8}]
                                0 root        -76    -     0B  1616K -        4 403:09   1.64% [kernel{if_io_tqg_4}]
                            54075 netdata      40   19   350M   213M nanslp  13  69:02   0.48% /usr/local/sbin/netdata -u netdata -P /var/db/netdata/netdata.pid{PLUGIN[freebsd]}
                            
                            

                            bce/bge is internet lines, so they are connecting converter (ISP devices). I can't check on ISP devices.
                            ix is LAN interface . It is connecting to cisco meraki switch, but I checked on this switch and I don't see any error related to interfaces (those connecting to Pfsense)

                            1 Reply Last reply Reply Quote 0
                            • S
                              stephenw10 Netgate Administrator
                              last edited by Feb 27, 2023, 1:55 AM

                              Hmm, hard to say.

                              You might swapping the ix and bce/bge NICs to see if the errors follow them. That might not be possible with the media types you have.

                              1 Reply Last reply Reply Quote 0
                              14 out of 14
                              • First post
                                14/14
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                This community forum collects and processes your personal information.
                                consent.not_received