Packet error on physical interface
-
Hi,
I am using Pfsense that was installed on Dell Server, and it's working on production.
But I got a lot of packet error on almost of interface, like bellow:[2.6.0-RELEASE][admin@pfs2]/root: netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll bce2 1500 <Link#3> b8:ac:6f:xx:xx:xx 949767111 889101 0 460586066 0 0 bce2 - fe80::%bce2/6 fe80::baac:6fff:f 0 - - 1 - - bce3* 1500 <Link#4> b8:ac:6f:97:06:f0 0 0 0 0 0 0 ix0 1500 <Link#5> 00:1b:21:xx:xx:xx 4708069675 4415248 0 11655678303 0 0 ix1 1500 <Link#6> 00:1b:21:xx:xx:xx 7296506329 12473470 0 11841483585 0 0for example: packet error rate for bce2 interface is: (889101/949767111)*100 = 0.9 %
This problem happens frequently, sometimes to bce2 interface, sometimes to other interfaces.
I am not sure this problem relate to hardware or just something wrong in setting of pfsense software.Anyone can help to give me some tips on troubleshooting for this problem ?
Thanks in advanceWilliam
-
The fact it's happening on all interfaces seems suspicious. The Intel ix NICs can report increased errors like that due to driver changes but I'm not aware of anything similar in bce.
Are they all linked to the same thing?You can look at the mac stats in the sysctl output to see what type of errors those are, at least for ix. For example:
dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.rx_missed_packets: 0 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.remote_faults: 0 dev.ix.0.mac_stats.local_faults: 1 dev.ix.0.mac_stats.short_discards: 0 -
@stephenw10 hi,
ix0 and ix1 are internal interfaces (LAN), and they were bonded into lagg0 interface.
bce* and bge* are inferfaces used for internet (WAN interface). they are configured PPPOE.
[2.6.0-RELEASE][admin@pfs2]/root: sysctl -a | grep mac_stats dev.ix.1.mac_stats.tx_frames_1024_1522: 11199336056 dev.ix.1.mac_stats.tx_frames_512_1023: 296554491 dev.ix.1.mac_stats.tx_frames_256_511: 267149362 dev.ix.1.mac_stats.tx_frames_128_255: 469563788 dev.ix.1.mac_stats.tx_frames_65_127: 1743645376 dev.ix.1.mac_stats.tx_frames_64: 200622859 dev.ix.1.mac_stats.management_pkts_txd: 0 dev.ix.1.mac_stats.mcast_pkts_txd: 113559 dev.ix.1.mac_stats.bcast_pkts_txd: 9892940 dev.ix.1.mac_stats.good_pkts_txd: 14176871932 dev.ix.1.mac_stats.total_pkts_txd: 14176872009 dev.ix.1.mac_stats.good_octets_txd: 16582022206821 dev.ix.1.mac_stats.checksum_errs: 16885390 dev.ix.1.mac_stats.management_pkts_drpd: 0 dev.ix.1.mac_stats.management_pkts_rcvd: 0 dev.ix.1.mac_stats.recv_jabberd: 0 dev.ix.1.mac_stats.recv_oversized: 0 dev.ix.1.mac_stats.recv_fragmented: 0 dev.ix.1.mac_stats.recv_undersized: 0 dev.ix.1.mac_stats.rx_frames_1024_1522: 1627134501 dev.ix.1.mac_stats.rx_frames_512_1023: 256050662 dev.ix.1.mac_stats.rx_frames_256_511: 201144253 dev.ix.1.mac_stats.rx_frames_128_255: 809553201 dev.ix.1.mac_stats.rx_frames_65_127: 5598262630 dev.ix.1.mac_stats.rx_frames_64: 218715925 dev.ix.1.mac_stats.bcast_pkts_rcvd: 120420301 dev.ix.1.mac_stats.mcast_pkts_rcvd: 111525769 dev.ix.1.mac_stats.good_pkts_rcvd: 8710861172 dev.ix.1.mac_stats.total_pkts_rcvd: 9968563894 dev.ix.1.mac_stats.good_octets_rcvd: 3170640992241 dev.ix.1.mac_stats.total_octets_rcvd: 3516919350290 dev.ix.1.mac_stats.xoff_recvd: 0 dev.ix.1.mac_stats.xoff_txd: 76 dev.ix.1.mac_stats.xon_recvd: 0 dev.ix.1.mac_stats.xon_txd: 1 dev.ix.1.mac_stats.rx_missed_packets: 2371 dev.ix.1.mac_stats.rec_len_errs: 0 dev.ix.1.mac_stats.remote_faults: 4 dev.ix.1.mac_stats.local_faults: 9775 dev.ix.1.mac_stats.short_discards: 0 dev.ix.1.mac_stats.byte_errs: 0 dev.ix.1.mac_stats.ill_errs: 0 dev.ix.1.mac_stats.crc_errs: 0 dev.ix.1.mac_stats.rx_errs: 16887761 dev.ix.0.mac_stats.tx_frames_1024_1522: 11117151076 dev.ix.0.mac_stats.tx_frames_512_1023: 282499408 dev.ix.0.mac_stats.tx_frames_256_511: 260685094 dev.ix.0.mac_stats.tx_frames_128_255: 462588596 dev.ix.0.mac_stats.tx_frames_65_127: 1733360311 dev.ix.0.mac_stats.tx_frames_64: 155798861 dev.ix.0.mac_stats.management_pkts_txd: 0 dev.ix.0.mac_stats.mcast_pkts_txd: 113549 dev.ix.0.mac_stats.bcast_pkts_txd: 11355561 dev.ix.0.mac_stats.good_pkts_txd: 14012083346 dev.ix.0.mac_stats.total_pkts_txd: 14012083485 dev.ix.0.mac_stats.good_octets_txd: 16449667332507 dev.ix.0.mac_stats.checksum_errs: 6506947 dev.ix.0.mac_stats.management_pkts_drpd: 0 dev.ix.0.mac_stats.management_pkts_rcvd: 0 dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.rx_frames_1024_1522: 1193629067 dev.ix.0.mac_stats.rx_frames_512_1023: 176745206 dev.ix.0.mac_stats.rx_frames_256_511: 145828688 dev.ix.0.mac_stats.rx_frames_128_255: 396491922 dev.ix.0.mac_stats.rx_frames_65_127: 3527593358 dev.ix.0.mac_stats.rx_frames_64: 196177107 dev.ix.0.mac_stats.bcast_pkts_rcvd: 987533 dev.ix.0.mac_stats.mcast_pkts_rcvd: 1256931 dev.ix.0.mac_stats.good_pkts_rcvd: 5636465348 dev.ix.0.mac_stats.total_pkts_rcvd: 5754406086 dev.ix.0.mac_stats.good_octets_rcvd: 2255966211057 dev.ix.0.mac_stats.total_octets_rcvd: 2306898407455 dev.ix.0.mac_stats.xoff_recvd: 0 dev.ix.0.mac_stats.xoff_txd: 137 dev.ix.0.mac_stats.xon_recvd: 0 dev.ix.0.mac_stats.xon_txd: 2 dev.ix.0.mac_stats.rx_missed_packets: 2312 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.remote_faults: 3 dev.ix.0.mac_stats.local_faults: 11548 dev.ix.0.mac_stats.short_discards: 0 dev.ix.0.mac_stats.byte_errs: 0 dev.ix.0.mac_stats.ill_errs: 0 dev.ix.0.mac_stats.crc_errs: 0 dev.ix.0.mac_stats.rx_errs: 6509259bce2 Interface:
[2.6.0-RELEASE][admin@pfs2]/root: sysctl -a | grep bce.2 dev.bce.2.com_no_buffers: 1723435 dev.bce.2.stat_CatchupInRuleCheckerP4Hit: 0 dev.bce.2.stat_CatchupInMBUFDiscards: 0 dev.bce.2.stat_CatchupInFTQDiscards: 0 dev.bce.2.stat_CatchupInRuleCheckerDiscards: 0 dev.bce.2.stat_IfInRuleCheckerP4Hit: 0 dev.bce.2.stat_IfInMBUFDiscards: 0 dev.bce.2.stat_IfInFTQDiscards: 0 dev.bce.2.stat_IfInRuleCheckerDiscards: 0 dev.bce.2.stat_IfInFramesL2FilterDiscards: 0 dev.bce.2.stat_XoffStateEntered: 0 dev.bce.2.stat_MacControlFramesReceived: 0 dev.bce.2.stat_FlowControlDone: 0 dev.bce.2.stat_OutXoffSent: 0 dev.bce.2.stat_OutXonSent: 0 dev.bce.2.stat_XoffPauseFramesReceived: 0 dev.bce.2.stat_XonPauseFramesReceived: 0 dev.bce.2.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0 dev.bce.2.stat_EtherStatsPktsTx1024Octetsto1522Octets: 94602036 dev.bce.2.stat_EtherStatsPktsTx512Octetsto1023Octets: 20576003 dev.bce.2.stat_EtherStatsPktsTx256Octetsto511Octets: 12982897 dev.bce.2.stat_EtherStatsPktsTx128Octetsto255Octets: 64841148 dev.bce.2.stat_EtherStatsPktsTx65Octetsto127Octets: 486941526 dev.bce.2.stat_EtherStatsPktsTx64Octets: 6797799 dev.bce.2.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0 dev.bce.2.stat_EtherStatsPktsRx1024Octetsto1522Octets: 1157838817 dev.bce.2.stat_EtherStatsPktsRx512Octetsto1023Octets: 27269494 dev.bce.2.stat_EtherStatsPktsRx256Octetsto511Octets: 24201015 dev.bce.2.stat_EtherStatsPktsRx128Octetsto255Octets: 42154606 dev.bce.2.stat_EtherStatsPktsRx65Octetsto127Octets: 152389240 dev.bce.2.stat_EtherStatsPktsRx64Octets: 7241567 dev.bce.2.stat_EtherStatsOversizePkts: 0 dev.bce.2.stat_EtherStatsUndersizePkts: 0 dev.bce.2.stat_EtherStatsJabbers: 0 dev.bce.2.stat_EtherStatsFragments: 0 dev.bce.2.stat_EtherStatsCollisions: 0 dev.bce.2.stat_Dot3StatsLateCollisions: 0 dev.bce.2.stat_Dot3StatsExcessiveCollisions: 0 dev.bce.2.stat_Dot3StatsDeferredTransmissions: 0 dev.bce.2.stat_Dot3StatsMultipleCollisionFrames: 0 dev.bce.2.stat_Dot3StatsSingleCollisionFrames: 0 dev.bce.2.stat_Dot3StatsAlignmentErrors: 0 dev.bce.2.stat_Dot3StatsFCSErrors: 0 dev.bce.2.stat_Dot3StatsCarrierSenseErrors: 0 dev.bce.2.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 dev.bce.2.stat_IfHCOutBroadcastPkts: 3 dev.bce.2.stat_IfHCOutMulticastPkts: 3 dev.bce.2.stat_IfHCOutUcastPkts: 686741403 dev.bce.2.stat_IfHCInBroadcastPkts: 0 dev.bce.2.stat_IfHCInMulticastPkts: 0 dev.bce.2.stat_IfHCInUcastPkts: 1411094739 dev.bce.2.stat_IfHCOutBadOctets: 0 dev.bce.2.stat_IfHCOutOctets: 203247572645 dev.bce.2.stat_IfHCInBadOctets: 0 dev.bce.2.stat_IfHcInOctets: 1701496006956 dev.bce.2.unexpected_attention_count: 0 dev.bce.2.dma_map_addr_tx_failed_count: 0 dev.bce.2.dma_map_addr_rx_failed_count: 0 dev.bce.2.mbuf_frag_count: 0 dev.bce.2.mbuf_alloc_failed_count: 0 dev.bce.2.l2fhdr_error_count: 0 dev.bce.2.%parent: pci2 dev.bce.2.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0235 class=0x020000 dev.bce.2.%location: slot=0 function=0 dbsf=pci0:2:0:0 dev.bce.2.%driver: bce dev.bce.2.%desc: QLogic NetXtreme II BCM5709 1000Base-T (C0) -
Hmm, well looks like they are all checksum errors on the ix NICs. However that was updated int he driver for 2.6 and then further changes were made more recently as some of those were shown to be spurious.
Looks it's buffer exhaustion on the bce NIC which normally means the NIC is not being serviced by the CPU fast enough for the incoming traffic.
I would check that value matched the current input error count from netstat though.Steve
-
@stephenw10
here is full output from sysctl and netstat:
output.txt -
Hmm, OK so those errors are shown as
InputDiscardsfor bge, ascom_no_buffersfor bce and aschecksum_errsin ix.Do you have flow control active on any of those links?
My first though here is that the NICs are just dropping packets because the CPU queues are maxed out.Steve
-
@stephenw10
Do you have flow control active on any of those links?
--> No, I don'tMy first though here is that the NICs are just dropping packets because the CPU queues are maxed out.
--> so how can I check CPU queues ? and Is there solution for this? -
Check the boot logs to make sure each NIC is coming up with the expected number of queues initially. Check the CPU usage per core on the firewall using
top -HaSPat the CLI. Do you have cores that are maxed out.
Try enabling flow control. That should prevent packet loss in the even the NIC cannot be serviced fast enough. Though it may introduce other issues. -
@stephenw10 ,
top -HaSPlast pid: 81614; load averages: 0.49, 0.57, 0.52 up 42+17:32:16 08:22:31 453 threads: 18 running, 384 sleeping, 51 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 2: 0.0% user, 0.0% nice, 16.7% system, 0.0% interrupt, 83.3% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 4: 0.0% user, 0.0% nice, 16.7% system, 0.0% interrupt, 83.3% idle CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 6: 0.0% user, 0.0% nice, 8.3% system, 0.0% interrupt, 91.7% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 8: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 10: 0.0% user, 0.0% nice, 7.7% system, 0.0% interrupt, 92.3% idle CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 106M Active, 6357M Inact, 1549M Wired, 763M Buf, 23G Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 256K CPU9 9 1022.9 100.00% [idle{idle: cpu9}] 11 root 155 ki31 0B 256K CPU11 11 1022.8 100.00% [idle{idle: cpu11}] 11 root 155 ki31 0B 256K CPU15 15 1022.6 100.00% [idle{idle: cpu15}] 11 root 155 ki31 0B 256K CPU7 7 1022.4 100.00% [idle{idle: cpu7}] 11 root 155 ki31 0B 256K CPU1 1 1021.9 100.00% [idle{idle: cpu1}] 11 root 155 ki31 0B 256K CPU8 8 1016.7 96.53% [idle{idle: cpu8}] 11 root 155 ki31 0B 256K CPU10 10 1016.9 96.35% [idle{idle: cpu10}] 11 root 155 ki31 0B 256K CPU12 12 1016.5 96.32% [idle{idle: cpu12}] 11 root 155 ki31 0B 256K CPU14 14 1016.7 95.39% [idle{idle: cpu14}] 11 root 155 ki31 0B 256K CPU0 0 1000.3 94.90% [idle{idle: cpu0}] 11 root 155 ki31 0B 256K CPU3 3 1021.4 91.30% [idle{idle: cpu3}] 11 root 155 ki31 0B 256K CPU4 4 1004.4 91.23% [idle{idle: cpu4}] 11 root 155 ki31 0B 256K RUN 6 958.7H 88.47% [idle{idle: cpu6}] 11 root 155 ki31 0B 256K CPU2 2 1010.2 85.78% [idle{idle: cpu2}] 11 root 155 ki31 0B 256K CPU13 13 1022.7 85.24% [idle{idle: cpu13}] 11 root 155 ki31 0B 256K CPU5 5 1021.5 63.86% [idle{idle: cpu5}] 0 root -92 - 0B 1616K - 2 420:57 8.32% [kernel{bge2 taskq}] 0 root -92 - 0B 1616K - 1 61.3H 8.04% [kernel{bge0 taskq}] 0 root -76 - 0B 1616K - 2 381:58 6.41% [kernel{if_io_tqg_2}] 0 root -92 - 0B 1616K - 4 357:50 4.62% [kernel{bge3 taskq}] 0 root -76 - 0B 1616K - 12 450:05 3.25% [kernel{if_io_tqg_12}] 0 root -76 - 0B 1616K - 0 541:11 2.73% [kernel{if_io_tqg_0}] 0 root -92 - 0B 1616K CPU0 0 906:30 2.41% [kernel{bge1 taskq}] 12 root -92 - 0B 816K WAIT 4 502:59 2.20% [intr{irq258: bce2}] 0 root -76 - 0B 1616K - 10 426:32 2.18% [kernel{if_io_tqg_10}] 0 root -76 - 0B 1616K - 4 384:42 2.01% [kernel{if_io_tqg_4}] 0 root -76 - 0B 1616K - 6 374:11 1.83% [kernel{if_io_tqg_6}] 81614 root 20 0 14M 4512K CPU6 6 0:00 1.74% top -HaSP 0 root -76 - 0B 1616K - 14 430:22 1.57% [kernel{if_io_tqg_14}] 0 root -76 - 0B 1616K - 8 440:59 1.49% [kernel{if_io_tqg_8}]cat dmesg.boot
ix0: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xecc0-0xecdf mem 0xdf300000-0xdf37ffff,0xdf2f8000-0xdf2fbfff irq 35 at device 0.0 on pci5 ix0: Using 2048 TX descriptors and 2048 RX descriptors ix0: Using 8 RX queues 8 TX queues ix0: Using MSI-X interrupts with 9 vectors ix0: allocated for 8 queues ix0: allocated for 8 rx queues ix0: Ethernet address: 00:1b:21:bc:77:c0 ix0: PCI Express Bus: Speed 5.0GT/s Width x4 ix0: eTrack 0x00012b2c PHY FW V65535 ix0: netmap queues/slots: TX 8/2048, RX 8/2048 ix1: <Intel(R) X520 82599ES (SFI/SFP+)> port 0xece0-0xecff mem 0xdf380000-0xdf3fffff,0xdf2fc000-0xdf2fffff irq 46 at device 0.1 on pci5 ix1: Using 2048 TX descriptors and 2048 RX descriptors ix1: Using 8 RX queues 8 TX queues ix1: Using MSI-X interrupts with 9 vectors ix1: allocated for 8 queues ix1: allocated for 8 rx queues ix1: Ethernet address: 00:1b:21:bc:77:c1 ix1: PCI Express Bus: Speed 5.0GT/s Width x4 ix1: eTrack 0x00012b2c PHY FW V65535 ix1: netmap queues/slots: TX 8/2048, RX 8/2048 pcib6: <ACPI PCI-PCI bridge> at device 7.0 on pci0 pci6: <ACPI PCI bus> on pcib6 pcib7: <ACPI PCI-PCI bridge> at device 9.0 on pci0 pci7: <ACPI PCI bus> on pcib7I find the ix0 and ix1 interface using 8 queues.
Additionally, I've re-checked and found pfsense already enabled flow control:
sysctl hw.ix
hw.ix.enable_rss: 1 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 3 hw.ix.max_interrupt_rate: 31250Thanks!!!
-
Flow control is negotiated though, both ends have to support it.
Nothing much happening in that top output. Was there any traffic flowing at that point? Were the errors increasing ?
What I expect to see is that when traffic through the NIC peaks the CPU core(s) servicing it max out and it drops packets. Assuming that is the cause.Are you able to see the number of queues on the bce/bge NICs?
vmstat -imay show it. -
@stephenw10
yes, a lot of traffic at that point, and the errors increasing as well.
CPU usage is always less than 10%.
[2.6.0-RELEASE][admin@pfs2]/root: vmstat -i interrupt total rate irq17: uhci0 24 0 irq19: ehci0 2947641 1 irq23: atapci0 2616920 1 cpu0:timer 1228218976 311 cpu1:timer 43088269 11 cpu2:timer 515777307 131 cpu3:timer 44479430 11 cpu4:timer 665350682 168 cpu5:timer 43841493 11 cpu6:timer 1046412756 265 cpu7:timer 41014489 10 cpu8:timer 163311089 41 cpu9:timer 42267453 11 cpu10:timer 143573306 36 cpu11:timer 43289875 11 cpu12:timer 162167544 41 cpu13:timer 44236981 11 cpu14:timer 152160861 39 cpu15:timer 45152052 11 irq257: bce1 6913402 2 irq258: bce2 2158281404 547 irq259: bce3 1 0 irq260: mfi0 7352305 2 irq261: ix0:rxq0 7415677330 1878 irq262: ix0:rxq1 709426817 180 irq263: ix0:rxq2 756092990 191 irq264: ix0:rxq3 701942814 178 irq265: ix0:rxq4 771849533 195 irq266: ix0:rxq5 709479026 180 irq267: ix0:rxq6 813804607 206 irq268: ix0:rxq7 701544714 178 irq269: ix0:aq 6 0 irq270: ix1:rxq0 7808112326 1977 irq271: ix1:rxq1 1041604395 264 irq272: ix1:rxq2 1028264841 260 irq273: ix1:rxq3 1045890761 265 irq274: ix1:rxq4 1094897663 277 irq275: ix1:rxq5 1009812406 256 irq276: ix1:rxq6 1165305943 295 irq277: ix1:rxq7 1047942990 265 irq278: ix1:aq 7 0 irq279: bge0 6303565889 1596 irq280: bge1 3249572863 823 irq281: bge2 1628365893 412 irq282: bge3 1149357028 291 Total 46754965102 11839 [2.6.0-RELEASE][admin@pfs2]/root: netstat -i Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll bce0* 1500 <Link#1> b8:ac:6f:97:06:ea 0 0 0 0 0 0 bce1 1500 <Link#2> b8:ac:6f:97:06:ec 7117685 0 0 70405 0 0 bce1 - fe80::%bce1/6 fe80::baac:6fff:f 0 - - 1 - - bce1 - 192.168.103.0 pfs2 2343399 - - 515 - - bce2 1500 <Link#3> b8:ac:6f:97:06:ee 1868223693 2155425 0 906688134 0 0 bce2 - fe80::%bce2/6 fe80::baac:6fff:f 0 - - 1 - - bce3 1500 <Link#4> b8:ac:6f:97:06:f0 0 0 0 0 0 0 bce3 - fe80::%bce3/6 fe80::baac:6fff:f 0 - - 1 - - bce3 - xxx.xxx.xx.xx xxx.xxx.xx.xx 0 - - 0 - - ix0 1500 <Link#5> 00:1b:21:bc:77:c0 6590339737 7248161 0 16393394382 0 0 ix1 1500 <Link#6> 00:1b:21:bc:77:c0 10209454971 22546676 0 16548545943 0 0 bge0 1500 <Link#7> 00:0a:f7:90:e6:ec 21040184018 25001666 0 8758689519 0 0 bge0 - fe80::%bge0/6 fe80::20a:f7ff:fe 0 - - 1 - - bge1 1500 <Link#8> 00:0a:f7:90:e6:ed 3801913176 6100037 0 2248351113 0 0 bge1 - fe80::%bge1/6 fe80::20a:f7ff:fe 0 - - 0 - - bge2 1500 <Link#9> 00:0a:f7:90:e6:ee 1722390896 1717684 0 825525005 0 0 bge2 - fe80::%bge2/6 fe80::20a:f7ff:fe 0 - - 1 - - bge3 1500 <Link#10> 00:0a:f7:90:e6:ef 1665701913 2291555 0 914404232 0 0 bge3 - fe80::%bge3/6 fe80::20a:f7ff:fe 0 - - 2 - - -
Hmm, looks like 1 irq per NIC for bce/bge. However I'd expect to see one CPU core at 100% if it was failing to service the NIC fast enough.
What are those NICs connected to? Are you seeing errors on the connected devices?
-
@stephenw10 ,
here is output of top:[2.6.0-RELEASE][admin@pfs2]/root: top -HaSP last pid: 11585; load averages: 0.58, 0.76, 0.67 up 45+17:40:30 08:30:45 450 threads: 17 running, 382 sleeping, 51 waiting CPU 0: 0.0% user, 0.0% nice, 12.8% system, 0.0% interrupt, 87.2% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 2: 0.0% user, 0.0% nice, 4.3% system, 0.0% interrupt, 95.7% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 4: 0.0% user, 0.0% nice, 8.0% system, 3.7% interrupt, 88.2% idle CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 6: 0.0% user, 0.0% nice, 10.7% system, 0.0% interrupt, 89.3% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 8: 0.0% user, 0.0% nice, 2.7% system, 0.0% interrupt, 97.3% idle CPU 9: 0.5% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.5% idle CPU 10: 0.5% user, 0.0% nice, 2.7% system, 0.0% interrupt, 96.8% idle CPU 11: 0.0% user, 0.0% nice, 0.5% system, 0.0% interrupt, 99.5% idle CPU 12: 0.0% user, 0.0% nice, 2.1% system, 0.0% interrupt, 97.9% idle CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 14: 0.0% user, 0.0% nice, 2.7% system, 0.0% interrupt, 97.3% idle CPU 15: 0.5% user, 0.0% nice, 0.5% system, 0.0% interrupt, 98.9% idle Mem: 104M Active, 6370M Inact, 1570M Wired, 782M Buf, 23G Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 256K CPU11 11 1094.8 100.00% [idle{idle: cpu11}] 11 root 155 ki31 0B 256K CPU13 13 1094.7 100.00% [idle{idle: cpu13}] 11 root 155 ki31 0B 256K CPU3 3 1093.3 99.84% [idle{idle: cpu3}] 11 root 155 ki31 0B 256K CPU1 1 1093.8 99.77% [idle{idle: cpu1}] 11 root 155 ki31 0B 256K CPU15 15 1094.6 99.46% [idle{idle: cpu15}] 11 root 155 ki31 0B 256K CPU7 7 1094.3 98.81% [idle{idle: cpu7}] 11 root 155 ki31 0B 256K CPU5 5 1093.5 98.73% [idle{idle: cpu5}] 11 root 155 ki31 0B 256K RUN 8 1088.5 98.21% [idle{idle: cpu8}] 11 root 155 ki31 0B 256K CPU14 14 1088.5 98.10% [idle{idle: cpu14}] 11 root 155 ki31 0B 256K CPU10 10 1088.6 97.99% [idle{idle: cpu10}] 11 root 155 ki31 0B 256K CPU12 12 1088.2 97.60% [idle{idle: cpu12}] 11 root 155 ki31 0B 256K CPU9 9 1094.8 97.09% [idle{idle: cpu9}] 11 root 155 ki31 0B 256K CPU2 2 1081.5 94.77% [idle{idle: cpu2}] 11 root 155 ki31 0B 256K CPU4 4 1075.0 92.37% [idle{idle: cpu4}] 11 root 155 ki31 0B 256K CPU0 0 1071.4 88.04% [idle{idle: cpu0}] 11 root 155 ki31 0B 256K CPU6 6 1028.6 85.34% [idle{idle: cpu6}] 0 root -92 - 0B 1616K - 6 63.1H 12.78% [kernel{bge0 taskq}] 0 root -92 - 0B 1616K - 0 938:32 8.50% [kernel{bge1 taskq}] 0 root -76 - 0B 1616K - 0 564:06 3.74% [kernel{if_io_tqg_0}] 12 root -92 - 0B 816K WAIT 4 533:51 3.09% [intr{irq258: bce2}] 0 root -92 - 0B 1616K - 2 446:02 2.95% [kernel{bge2 taskq}] 0 root -92 - 0B 1616K - 4 395:52 2.95% [kernel{bge3 taskq}] 0 root -76 - 0B 1616K - 12 467:41 2.23% [kernel{if_io_tqg_12}] 0 root -76 - 0B 1616K - 6 389:13 2.03% [kernel{if_io_tqg_6}] 0 root -76 - 0B 1616K - 2 398:09 2.03% [kernel{if_io_tqg_2}] 0 root -76 - 0B 1616K - 10 442:43 1.90% [kernel{if_io_tqg_10}] 0 root -76 - 0B 1616K - 14 447:08 1.84% [kernel{if_io_tqg_14}] 0 root -76 - 0B 1616K - 8 457:22 1.69% [kernel{if_io_tqg_8}] 0 root -76 - 0B 1616K - 4 403:09 1.64% [kernel{if_io_tqg_4}] 54075 netdata 40 19 350M 213M nanslp 13 69:02 0.48% /usr/local/sbin/netdata -u netdata -P /var/db/netdata/netdata.pid{PLUGIN[freebsd]}bce/bge is internet lines, so they are connecting converter (ISP devices). I can't check on ISP devices.
ix is LAN interface . It is connecting to cisco meraki switch, but I checked on this switch and I don't see any error related to interfaces (those connecting to Pfsense) -
Hmm, hard to say.
You might swapping the ix and bce/bge NICs to see if the errors follow them. That might not be possible with the media types you have.
