Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Errors all over the place

    Scheduled Pinned Locked Moved General pfSense Questions
    6 Posts 2 Posters 3.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I
      InQuize
      last edited by

      Hello.
      Originally, my setup was functioning great, it was a standalone box. But then I went and merged it with FreeNAS box using ESXi 6.0 and hardware passthrough for my NICs and HBA. Everything seems working fine for about a month now, but I started getting IN errors on WAN interface and they continue incrementing..

      LAN is a LACP LAGG of 2 ports on fiber igb card connected to switch, which breaks up into serveral VLANs. Works fine exept very few OUT errors caused by LACP itself when systems are up but OSes are not, eg. in BIOS, when LACP is not initialized yet.

      WAN is a simple Intel Desktop CT adapter, so em driver.
      In front there is an ONT modem, which "does not support bridge mode" accroding to my ISP, so WAN space is another LAN..

      All offloading options are disabled.

      Here is interface status:

      Same count in "dev.em.0.mac_stats.missed_packets: 4023674":

      $ sysctl dev.em
      dev.em.%parent: 
      dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.4.2
      dev.em.0.%driver: em
      dev.em.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE60.S1F0
      dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000
      dev.em.0.%parent: pci19
      dev.em.0.nvm: -1
      dev.em.0.debug: -1
      dev.em.0.fc: 3
      dev.em.0.rx_int_delay: 0
      dev.em.0.tx_int_delay: 66
      dev.em.0.rx_abs_int_delay: 66
      dev.em.0.tx_abs_int_delay: 66
      dev.em.0.itr: 488
      dev.em.0.rx_processing_limit: 100
      dev.em.0.eee_control: 1
      dev.em.0.link_irq: 0
      dev.em.0.mbuf_alloc_fail: 0
      dev.em.0.cluster_alloc_fail: 0
      dev.em.0.dropped: 0
      dev.em.0.tx_dma_fail: 0
      dev.em.0.rx_overruns: 196259
      dev.em.0.watchdog_timeouts: 0
      dev.em.0.device_control: 1477444168
      dev.em.0.rx_control: 67141634
      dev.em.0.fc_high_water: 18432
      dev.em.0.fc_low_water: 16932
      dev.em.0.queue0.txd_head: 793
      dev.em.0.queue0.txd_tail: 793
      dev.em.0.queue0.tx_irq: 0
      dev.em.0.queue0.no_desc_avail: 0
      dev.em.0.queue0.rxd_head: 624
      dev.em.0.queue0.rxd_tail: 623
      dev.em.0.queue0.rx_irq: 0
      dev.em.0.mac_stats.excess_coll: 0
      dev.em.0.mac_stats.single_coll: 0
      dev.em.0.mac_stats.multiple_coll: 0
      dev.em.0.mac_stats.late_coll: 0
      dev.em.0.mac_stats.collision_count: 0
      dev.em.0.mac_stats.symbol_errors: 0
      dev.em.0.mac_stats.sequence_errors: 0
      dev.em.0.mac_stats.defer_count: 0
      dev.em.0.mac_stats.missed_packets: 4023674
      dev.em.0.mac_stats.recv_no_buff: 1147
      dev.em.0.mac_stats.recv_undersize: 0
      dev.em.0.mac_stats.recv_fragmented: 0
      dev.em.0.mac_stats.recv_oversize: 0
      dev.em.0.mac_stats.recv_jabber: 0
      dev.em.0.mac_stats.recv_errs: 0
      dev.em.0.mac_stats.crc_errs: 0
      dev.em.0.mac_stats.alignment_errs: 0
      dev.em.0.mac_stats.coll_ext_errs: 0
      dev.em.0.mac_stats.xon_recvd: 0
      dev.em.0.mac_stats.xon_txd: 229889
      dev.em.0.mac_stats.xoff_recvd: 0
      dev.em.0.mac_stats.xoff_txd: 4253562
      dev.em.0.mac_stats.total_pkts_recvd: 1019190241
      dev.em.0.mac_stats.good_pkts_recvd: 1015166567
      dev.em.0.mac_stats.bcast_pkts_recvd: 43766
      dev.em.0.mac_stats.mcast_pkts_recvd: 37032
      dev.em.0.mac_stats.rx_frames_64: 37869404
      dev.em.0.mac_stats.rx_frames_65_127: 46904834
      dev.em.0.mac_stats.rx_frames_128_255: 70985248
      dev.em.0.mac_stats.rx_frames_256_511: 29553892
      dev.em.0.mac_stats.rx_frames_512_1023: 74673383
      dev.em.0.mac_stats.rx_frames_1024_1522: 755179806
      dev.em.0.mac_stats.good_octets_recvd: 1184386034957
      dev.em.0.mac_stats.good_octets_txd: 150188980069
      dev.em.0.mac_stats.total_pkts_txd: 811219743
      dev.em.0.mac_stats.good_pkts_txd: 806736292
      dev.em.0.mac_stats.bcast_pkts_txd: 120
      dev.em.0.mac_stats.mcast_pkts_txd: 11
      dev.em.0.mac_stats.tx_frames_64: 161896304
      dev.em.0.mac_stats.tx_frames_65_127: 507949485
      dev.em.0.mac_stats.tx_frames_128_255: 59335118
      dev.em.0.mac_stats.tx_frames_256_511: 15082844
      dev.em.0.mac_stats.tx_frames_512_1023: 4651980
      dev.em.0.mac_stats.tx_frames_1024_1522: 57820561
      dev.em.0.mac_stats.tso_txd: 0
      dev.em.0.mac_stats.tso_ctx_fail: 0
      dev.em.0.interrupts.asserts: 832965679
      dev.em.0.interrupts.rx_pkt_timer: 73507
      dev.em.0.interrupts.rx_abs_timer: 0
      dev.em.0.interrupts.tx_pkt_timer: 31105
      dev.em.0.interrupts.tx_abs_timer: 40799
      dev.em.0.interrupts.tx_queue_empty: 0
      dev.em.0.interrupts.tx_queue_min_thresh: 0
      dev.em.0.interrupts.rx_desc_min_thresh: 0
      dev.em.0.interrupts.rx_overrun: 77
      dev.em.0.wake: 0
      

      I've swapped cable with no luck. NIC itself I believe is fine too since it all started after moving into virtual enviromnent. I have 2 more NICs alike to test, but would like to avoid downtime needed for hadware swap if possible.

      So what are these 'missed packets'? And should I be worried?
      Googling gave no answers.

      1 Reply Last reply Reply Quote 0
      • I
        InQuize
        last edited by

        During last 8 hours there were about 40k new errors under load (79 Mb/s on 95th percentile)… Anyone?

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          
          dev.em.0.mac_stats.missed_packets: 4023674
          dev.em.0.mac_stats.recv_no_buff: 1147
          

          Looks like the packets are coming in too fast for the NIC/driver/OS to handle. What version of pfSense? Are you sure the hardware is not being overloaded?

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • I
            InQuize
            last edited by

            I highly doubt it. I mean, I'm far from being good at networking specifics, and may only guess about reasons of such issue as mine, so I would rather provide more experienced people with as much raw data as I can. At least it feels like the most efficient path I can take in that situation. I appologise for that pile of information in advance, but setup is somewhat sophisticated.

            Exact specs for ESXi host running only pfSense and FreeNAS are:
            ►CPU: Intel Xeon E3-1220V3
            ►MB: Supermicro X10SLM-F
            ►RAM: 32 GB KVR16E11/8
            ►HDD: HDS721050CLA362
            ►Box: Fractal Design Define R4 Black
            ►PSU: Chieftec GPS-700A8 @ 100W
            ►UPS: APC Back-UPS BC650-RS
            pfSense passthrough:
            ►WAN: Intel EXPI9301CT
            ►Trunk: Intel E1G42EFBLK
            FreeNAS passthrough & spinning rust:
            ►HBA: IBM ServeRAID M1115 [LSI9211-8i fw.20 IT w/o bios]
            ►'RAID1': 3x 2TB [WD20EARS + WD20EZRX + WD2001FASS]
            ►'JBOD': 3x 4TB HDS5C4040ALE630 + 4TB WD40EZRX

            pfSense packages: squid, Lightsquid, Service Watchdog, nut, OpenVPN Client Export Utility, Open-VM-Tools + frequent use of OpenVPN RA Server for traffic redirection and data sync
            FreeNAS: Syncthing live backups form Windows machines across the network, couple Plex jails, NFS/SMB shared database and data storage for Proxmox node (OwnCloud, p2p, etc), lazy compression, snapshoting, none of hard stuff like ZFS deduplication or encryption..

            pfSense & FreeNAS are on latest stable release versions, ESXi is at 6.0.0, 3029758

            And it's a home host serving only as high load as members of average sized family can produce..

            CPU is idle most of the time (experiences full frequency loads probably only at night while generating video preview thumbnails for Plex's new media. And even for 100% utilization scenario I made CPU reservation of 1000Mhz for pfSense VM).

            RAM: 4GB pfSense (50% allowed for & often used by squid) / 23GB FreeNAS / rest reserved by ESXi.

            MBUF Usage is most of the time at 20%, not sure whether it can spike to a serious amount, never noticed..

            WAN throughput from ISP is 100 mbit bidirectional with 200 mbit Rx burst.
            Typical load - judge yourself: RRD graphs

            By my expectations, that host should be more than capable of my tasks if not overkill (except in RAM for ZFS, it's on border line, but I don't plan to expand on that particular platform). What's more important, I consider that system built using all crucial rules for both OSes, although not running on bare metal.
            Correct me if I'm wrong.

            1 Reply Last reply Reply Quote 0
            • I
              InQuize
              last edited by

              New details.
              While monitoring the system today I was looking at the interface statistics, wondering what makes diference between WAN and other interfaces. Because at least two of them experience equal or higher load than WAN. Then I realized that I'm not seeing the whole picture because other stats are for virtual (VLAN) interfaces rather than physical as em0 for WAN. So pfSense accounts for errors to send out packets from these interfaces but can not divide actual error data on physical NIC for each virtual interface..

              $ sysctl dev.igb

              
              dev.igb.%parent: 
              dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
              dev.igb.0.%driver: igb
              dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0
              dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
              dev.igb.0.%parent: pci3
              dev.igb.0.nvm: -1
              dev.igb.0.enable_aim: 1
              dev.igb.0.fc: 3
              dev.igb.0.rx_processing_limit: 100
              dev.igb.0.link_irq: 0
              dev.igb.0.dropped: 0
              dev.igb.0.tx_dma_fail: 0
              dev.igb.0.rx_overruns: 404
              dev.igb.0.watchdog_timeouts: 0
              dev.igb.0.device_control: 1087117889
              dev.igb.0.rx_control: 67141634
              dev.igb.0.interrupt_mask: 157
              dev.igb.0.extended_int_mask: 2147483648
              dev.igb.0.tx_buf_alloc: 0
              dev.igb.0.rx_buf_alloc: 0
              dev.igb.0.fc_high_water: 58976
              dev.igb.0.fc_low_water: 58960
              dev.igb.0.queue0.no_desc_avail: 1261891
              dev.igb.0.queue0.tx_packets: 1313009825
              dev.igb.0.queue0.rx_packets: 781759341
              dev.igb.0.queue0.rx_bytes: 580723340999
              dev.igb.0.queue0.lro_queued: 0
              dev.igb.0.queue0.lro_flushed: 0
              dev.igb.0.mac_stats.excess_coll: 0
              dev.igb.0.mac_stats.single_coll: 0
              dev.igb.0.mac_stats.multiple_coll: 0
              dev.igb.0.mac_stats.late_coll: 0
              dev.igb.0.mac_stats.collision_count: 0
              dev.igb.0.mac_stats.symbol_errors: 0
              dev.igb.0.mac_stats.sequence_errors: 0
              dev.igb.0.mac_stats.defer_count: 0
              dev.igb.0.mac_stats.missed_packets: 83631
              dev.igb.0.mac_stats.recv_no_buff: 25953
              dev.igb.0.mac_stats.recv_undersize: 0
              dev.igb.0.mac_stats.recv_fragmented: 0
              dev.igb.0.mac_stats.recv_oversize: 0
              dev.igb.0.mac_stats.recv_jabber: 0
              dev.igb.0.mac_stats.recv_errs: 0
              dev.igb.0.mac_stats.crc_errs: 0
              dev.igb.0.mac_stats.alignment_errs: 0
              dev.igb.0.mac_stats.coll_ext_errs: 0
              dev.igb.0.mac_stats.xon_recvd: 0
              dev.igb.0.mac_stats.xon_txd: 0
              dev.igb.0.mac_stats.xoff_recvd: 0
              dev.igb.0.mac_stats.xoff_txd: 0
              dev.igb.0.mac_stats.total_pkts_recvd: 782461121
              dev.igb.0.mac_stats.good_pkts_recvd: 781759301
              dev.igb.0.mac_stats.bcast_pkts_recvd: 1520864
              dev.igb.0.mac_stats.mcast_pkts_recvd: 203559
              dev.igb.0.mac_stats.rx_frames_64: 81560000
              dev.igb.0.mac_stats.rx_frames_65_127: 277073969
              dev.igb.0.mac_stats.rx_frames_128_255: 43744716
              dev.igb.0.mac_stats.rx_frames_256_511: 15270084
              dev.igb.0.mac_stats.rx_frames_512_1023: 4240271
              dev.igb.0.mac_stats.rx_frames_1024_1522: 359870261
              dev.igb.0.mac_stats.good_octets_recvd: 586977128523
              dev.igb.0.mac_stats.good_octets_txd: 1351374080382
              dev.igb.0.mac_stats.total_pkts_txd: 1313009811
              dev.igb.0.mac_stats.good_pkts_txd: 1313009811
              dev.igb.0.mac_stats.bcast_pkts_txd: 30114
              dev.igb.0.mac_stats.mcast_pkts_txd: 1609000
              dev.igb.0.mac_stats.tx_frames_64: 16842087
              dev.igb.0.mac_stats.tx_frames_65_127: 336263799
              dev.igb.0.mac_stats.tx_frames_128_255: 41689908
              dev.igb.0.mac_stats.tx_frames_256_511: 18165668
              dev.igb.0.mac_stats.tx_frames_512_1023: 36777014
              dev.igb.0.mac_stats.tx_frames_1024_1522: 863271335
              dev.igb.0.mac_stats.tso_txd: 0
              dev.igb.0.mac_stats.tso_ctx_fail: 0
              dev.igb.0.interrupts.asserts: 1458531356
              dev.igb.0.interrupts.rx_pkt_timer: 781750145
              dev.igb.0.interrupts.rx_abs_timer: 0
              dev.igb.0.interrupts.tx_pkt_timer: 0
              dev.igb.0.interrupts.tx_abs_timer: 781754685
              dev.igb.0.interrupts.tx_queue_empty: 1312992017
              dev.igb.0.interrupts.tx_queue_min_thresh: 0
              dev.igb.0.interrupts.rx_desc_min_thresh: 0
              dev.igb.0.interrupts.rx_overrun: 0
              dev.igb.0.host.breaker_tx_pkt: 0
              dev.igb.0.host.host_tx_pkt_discard: 0
              dev.igb.0.host.rx_pkt: 9156
              dev.igb.0.host.breaker_rx_pkts: 0
              dev.igb.0.host.breaker_rx_pkt_drop: 0
              dev.igb.0.host.tx_good_pkt: 17794
              dev.igb.0.host.breaker_tx_pkt_drop: 0
              dev.igb.0.host.rx_good_bytes: 586977148059
              dev.igb.0.host.tx_good_bytes: 1351374080382
              dev.igb.0.host.length_errors: 0
              dev.igb.0.host.serdes_violation_pkt: 0
              dev.igb.0.host.header_redir_missed: 0
              dev.igb.0.wake: 0
              dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
              dev.igb.1.%driver: igb
              dev.igb.1.%location: slot=0 function=1
              dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
              dev.igb.1.%parent: pci3
              dev.igb.1.nvm: -1
              dev.igb.1.enable_aim: 1
              dev.igb.1.fc: 3
              dev.igb.1.rx_processing_limit: 100
              dev.igb.1.link_irq: 0
              dev.igb.1.dropped: 0
              dev.igb.1.tx_dma_fail: 0
              dev.igb.1.rx_overruns: 7
              dev.igb.1.watchdog_timeouts: 0
              dev.igb.1.device_control: 1087117889
              dev.igb.1.rx_control: 67141634
              dev.igb.1.interrupt_mask: 157
              dev.igb.1.extended_int_mask: 2147483648
              dev.igb.1.tx_buf_alloc: 0
              dev.igb.1.rx_buf_alloc: 0
              dev.igb.1.fc_high_water: 58976
              dev.igb.1.fc_low_water: 58960
              dev.igb.1.queue0.no_desc_avail: 26818
              dev.igb.1.queue0.tx_packets: 1312415919
              dev.igb.1.queue0.rx_packets: 784960625
              dev.igb.1.queue0.rx_bytes: 657958891610
              dev.igb.1.queue0.lro_queued: 0
              dev.igb.1.queue0.lro_flushed: 0
              dev.igb.1.mac_stats.excess_coll: 0
              dev.igb.1.mac_stats.single_coll: 0
              dev.igb.1.mac_stats.multiple_coll: 0
              dev.igb.1.mac_stats.late_coll: 0
              dev.igb.1.mac_stats.collision_count: 0
              dev.igb.1.mac_stats.symbol_errors: 0
              dev.igb.1.mac_stats.sequence_errors: 0
              dev.igb.1.mac_stats.defer_count: 0
              dev.igb.1.mac_stats.missed_packets: 1074
              dev.igb.1.mac_stats.recv_no_buff: 280
              dev.igb.1.mac_stats.recv_undersize: 0
              dev.igb.1.mac_stats.recv_fragmented: 0
              dev.igb.1.mac_stats.recv_oversize: 0
              dev.igb.1.mac_stats.recv_jabber: 0
              dev.igb.1.mac_stats.recv_errs: 0
              dev.igb.1.mac_stats.crc_errs: 0
              dev.igb.1.mac_stats.alignment_errs: 0
              dev.igb.1.mac_stats.coll_ext_errs: 0
              dev.igb.1.mac_stats.xon_recvd: 0
              dev.igb.1.mac_stats.xon_txd: 0
              dev.igb.1.mac_stats.xoff_recvd: 0
              dev.igb.1.mac_stats.xoff_txd: 0
              dev.igb.1.mac_stats.total_pkts_recvd: 1168917353
              dev.igb.1.mac_stats.good_pkts_recvd: 784960599
              dev.igb.1.mac_stats.bcast_pkts_recvd: 405858
              dev.igb.1.mac_stats.mcast_pkts_recvd: 294354
              dev.igb.1.mac_stats.rx_frames_64: 89494759
              dev.igb.1.mac_stats.rx_frames_65_127: 232942520
              dev.igb.1.mac_stats.rx_frames_128_255: 35599853
              dev.igb.1.mac_stats.rx_frames_256_511: 8289293
              dev.igb.1.mac_stats.rx_frames_512_1023: 3432957
              dev.igb.1.mac_stats.rx_frames_1024_1522: 415201217
              dev.igb.1.mac_stats.good_octets_recvd: 664238288288
              dev.igb.1.mac_stats.good_octets_txd: 1161071633488
              dev.igb.1.mac_stats.total_pkts_txd: 1312415798
              dev.igb.1.mac_stats.good_pkts_txd: 1312415798
              dev.igb.1.mac_stats.bcast_pkts_txd: 197034
              dev.igb.1.mac_stats.mcast_pkts_txd: 1608910
              dev.igb.1.mac_stats.tx_frames_64: 263810305
              dev.igb.1.mac_stats.tx_frames_65_127: 201290334
              dev.igb.1.mac_stats.tx_frames_128_255: 57384858
              dev.igb.1.mac_stats.tx_frames_256_511: 21661731
              dev.igb.1.mac_stats.tx_frames_512_1023: 47026530
              dev.igb.1.mac_stats.tx_frames_1024_1522: 721242040
              dev.igb.1.mac_stats.tso_txd: 0
              dev.igb.1.mac_stats.tso_ctx_fail: 0
              dev.igb.1.interrupts.asserts: 1354951558
              dev.igb.1.interrupts.rx_pkt_timer: 784951616
              dev.igb.1.interrupts.rx_abs_timer: 0
              dev.igb.1.interrupts.tx_pkt_timer: 0
              dev.igb.1.interrupts.tx_abs_timer: 784960591
              dev.igb.1.interrupts.tx_queue_empty: 1312400215
              dev.igb.1.interrupts.tx_queue_min_thresh: 0
              dev.igb.1.interrupts.rx_desc_min_thresh: 0
              dev.igb.1.interrupts.rx_overrun: 0
              dev.igb.1.host.breaker_tx_pkt: 0
              dev.igb.1.host.host_tx_pkt_discard: 0
              dev.igb.1.host.rx_pkt: 8983
              dev.igb.1.host.breaker_rx_pkts: 0
              dev.igb.1.host.breaker_rx_pkt_drop: 0
              dev.igb.1.host.tx_good_pkt: 15583
              dev.igb.1.host.breaker_tx_pkt_drop: 0
              dev.igb.1.host.rx_good_bytes: 664238306626
              dev.igb.1.host.tx_good_bytes: 1161071633488
              dev.igb.1.host.length_errors: 0
              dev.igb.1.host.serdes_violation_pkt: 0
              dev.igb.1.host.header_redir_missed: 0
              
              

              dev.igb.0.mac_stats.missed_packets: 83631
              dev.igb.0.mac_stats.recv_no_buff: 25953
              dev.igb.1.mac_stats.missed_packets: 1074
              dev.igb.1.mac_stats.recv_no_buff: 280

              So the problem is system wide  :( or not really since VMXNET3 adapter is pretty clean?

              Also, using simple math, there was about 340k new errors on WAN during last 23 hours. That's ~15k per hour, which is 3 times more than during 8 hours period. Load during 23h was approx 17Mb/s vs 79Mb/s during 8h (95th percentile according to RRD).

              So issue should hardly be dependent on load, right?

              1 Reply Last reply Reply Quote 0
              • I
                InQuize
                last edited by

                Well, I made a progress on incoming errors..
                These errors accumulated rapidly and while doing no observable harm. But about a week ago situation became worse. On a month of uptime system freaked out, error counter went through the roof and WAN went down. I rebooted.. And it repeated 3 times during 1st or 2nd 24 hours of uptime since then (under load). And that's when I started playing with hardware and poking drivers with a stick. So, the results:

                • EXPI9301CT NIC itself is working fine, I tried swapping with the same exact model - same problems. Missed packets and "recv_no_buff" counters. Even on igb card, while this one is installed. So it should be driver (em) or this specific model.

                • Changing model, at first I used I217-LM (one of onboard NICs, em driver) for WAN passthrough. According to vSphere Client it has own pci lane, unlike other one, which seems to be interconnected through PCH. "That one should be superior," - I thought - "as long as they are pretty similar looking on Intel ARK". As a result, there was no incrementing error counters from the start on any interface, but during 2nd 24 hours of "stressing" the system with 80 mbps on average, WAN timed out for couple seconds (I actually experienced it), interface dropped 923 errors and continued functioning. Flimsy.. Although no errors on dev.igb. So, again should be driver (em) or this specific model (if momentary glitch was unrelated)

                • Then I switched WAN to another onboard NIC - I210 (igb driver). No incoming errors, no missed packets. Same on fiber card. I pushed roughly the same amount of traffic through system in 2 days which initially brought my WAN down after the month of runtime. System still stable. Missed packets and "recv_no_buff" at 0 on all NICs.

                To sum up, I still don't know whether it's the specific model that causes issue during passthrough (while it is actually functional and now acts as a secondary LACP port for ESXi management in place of onboard one that is now passed to pfSense for WAN) or the em driver and there could be other em NICs having same issue with passthrough or even the 82574L chip has something to do in hardware. EXPI9301CT is a very old model meant for desktops, not virtual environments, although it is still present in Intel's 2016 catalogue. And also given that em dirver in FreeBSD man pages is dated October 11, 2011 while igb is of March 25, 2013, I just made a conclusion for myself that later is just more stable and Intel NICs are not bulletproof as I thought.

                I don't know what are these outgoing errors.. I recall having them all the time, even on bare metal, even on other platform. There is literally no info I managed to find about any possible causes of them or at least the meaning of these counters, so if someone may enlighten me, please do.. At least, they seem to not be as harmless.

                $ sysctl dev.igb

                
                dev.igb.%parent: 
                dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
                dev.igb.0.%driver: igb
                dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0
                dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 class=0x020000
                dev.igb.0.%parent: pci3
                dev.igb.0.nvm: -1
                dev.igb.0.enable_aim: 1
                dev.igb.0.fc: 3
                dev.igb.0.rx_processing_limit: 100
                dev.igb.0.dmac: 0
                dev.igb.0.eee_disabled: 0
                dev.igb.0.link_irq: 0
                dev.igb.0.dropped: 0
                dev.igb.0.tx_dma_fail: 0
                dev.igb.0.rx_overruns: 0
                dev.igb.0.watchdog_timeouts: 0
                dev.igb.0.device_control: 1478230593
                dev.igb.0.rx_control: 71335938
                dev.igb.0.interrupt_mask: 157
                dev.igb.0.extended_int_mask: 2147483648
                dev.igb.0.tx_buf_alloc: 0
                dev.igb.0.rx_buf_alloc: 0
                dev.igb.0.fc_high_water: 31328
                dev.igb.0.fc_low_water: 31312
                dev.igb.0.queue0.no_desc_avail: 0
                dev.igb.0.queue0.tx_packets: 620869912
                dev.igb.0.queue0.rx_packets: 618707893
                dev.igb.0.queue0.rx_bytes: 851127332692
                dev.igb.0.queue0.lro_queued: 0
                dev.igb.0.queue0.lro_flushed: 0
                dev.igb.0.mac_stats.excess_coll: 0
                dev.igb.0.mac_stats.single_coll: 0
                dev.igb.0.mac_stats.multiple_coll: 0
                dev.igb.0.mac_stats.late_coll: 0
                dev.igb.0.mac_stats.collision_count: 0
                dev.igb.0.mac_stats.symbol_errors: 0
                dev.igb.0.mac_stats.sequence_errors: 0
                dev.igb.0.mac_stats.defer_count: 0
                dev.igb.0.mac_stats.missed_packets: 0
                dev.igb.0.mac_stats.recv_no_buff: 0
                dev.igb.0.mac_stats.recv_undersize: 0
                dev.igb.0.mac_stats.recv_fragmented: 0
                dev.igb.0.mac_stats.recv_oversize: 0
                dev.igb.0.mac_stats.recv_jabber: 0
                dev.igb.0.mac_stats.recv_errs: 0
                dev.igb.0.mac_stats.crc_errs: 0
                dev.igb.0.mac_stats.alignment_errs: 0
                dev.igb.0.mac_stats.coll_ext_errs: 0
                dev.igb.0.mac_stats.xon_recvd: 0
                dev.igb.0.mac_stats.xon_txd: 0
                dev.igb.0.mac_stats.xoff_recvd: 0
                dev.igb.0.mac_stats.xoff_txd: 0
                dev.igb.0.mac_stats.total_pkts_recvd: 618715218
                dev.igb.0.mac_stats.good_pkts_recvd: 618707618
                dev.igb.0.mac_stats.bcast_pkts_recvd: 1428
                dev.igb.0.mac_stats.mcast_pkts_recvd: 3090
                dev.igb.0.mac_stats.rx_frames_64: 601133
                dev.igb.0.mac_stats.rx_frames_65_127: 13638243
                dev.igb.0.mac_stats.rx_frames_128_255: 10436929
                dev.igb.0.mac_stats.rx_frames_256_511: 9534739
                dev.igb.0.mac_stats.rx_frames_512_1023: 13436838
                dev.igb.0.mac_stats.rx_frames_1024_1522: 571059736
                dev.igb.0.mac_stats.good_octets_recvd: 853601812367
                dev.igb.0.mac_stats.good_octets_txd: 71259681700
                dev.igb.0.mac_stats.total_pkts_txd: 620869651
                dev.igb.0.mac_stats.good_pkts_txd: 620869651
                dev.igb.0.mac_stats.bcast_pkts_txd: 3
                dev.igb.0.mac_stats.mcast_pkts_txd: 3
                dev.igb.0.mac_stats.tx_frames_64: 3612420
                dev.igb.0.mac_stats.tx_frames_65_127: 560906135
                dev.igb.0.mac_stats.tx_frames_128_255: 28742303
                dev.igb.0.mac_stats.tx_frames_256_511: 8907879
                dev.igb.0.mac_stats.tx_frames_512_1023: 4722579
                dev.igb.0.mac_stats.tx_frames_1024_1522: 13978335
                dev.igb.0.mac_stats.tso_txd: 0
                dev.igb.0.mac_stats.tso_ctx_fail: 0
                dev.igb.0.interrupts.asserts: 965490770
                dev.igb.0.interrupts.rx_pkt_timer: 618687254
                dev.igb.0.interrupts.rx_abs_timer: 0
                dev.igb.0.interrupts.tx_pkt_timer: 0
                dev.igb.0.interrupts.tx_abs_timer: 0
                dev.igb.0.interrupts.tx_queue_empty: 620853924
                dev.igb.0.interrupts.tx_queue_min_thresh: 618707618
                dev.igb.0.interrupts.rx_desc_min_thresh: 0
                dev.igb.0.interrupts.rx_overrun: 0
                dev.igb.0.host.breaker_tx_pkt: 0
                dev.igb.0.host.host_tx_pkt_discard: 0
                dev.igb.0.host.rx_pkt: 20364
                dev.igb.0.host.breaker_rx_pkts: 0
                dev.igb.0.host.breaker_rx_pkt_drop: 0
                dev.igb.0.host.tx_good_pkt: 15727
                dev.igb.0.host.breaker_tx_pkt_drop: 0
                dev.igb.0.host.rx_good_bytes: 853601838827
                dev.igb.0.host.tx_good_bytes: 71259681700
                dev.igb.0.host.length_errors: 0
                dev.igb.0.host.serdes_violation_pkt: 0
                dev.igb.0.host.header_redir_missed: 0
                dev.igb.0.wake: 0
                dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
                dev.igb.1.%driver: igb
                dev.igb.1.%location: slot=0 function=0 handle=\_SB_.PCI0.PE50.S1F0
                dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
                dev.igb.1.%parent: pci11
                dev.igb.1.nvm: -1
                dev.igb.1.enable_aim: 1
                dev.igb.1.fc: 3
                dev.igb.1.rx_processing_limit: 100
                dev.igb.1.link_irq: 0
                dev.igb.1.dropped: 0
                dev.igb.1.tx_dma_fail: 0
                dev.igb.1.rx_overruns: 0
                dev.igb.1.watchdog_timeouts: 0
                dev.igb.1.device_control: 1087117889
                dev.igb.1.rx_control: 67141634
                dev.igb.1.interrupt_mask: 157
                dev.igb.1.extended_int_mask: 2147483648
                dev.igb.1.tx_buf_alloc: 0
                dev.igb.1.rx_buf_alloc: 0
                dev.igb.1.fc_high_water: 58976
                dev.igb.1.fc_low_water: 58960
                dev.igb.1.queue0.no_desc_avail: 3
                dev.igb.1.queue0.tx_packets: 1749306695
                dev.igb.1.queue0.rx_packets: 631652753
                dev.igb.1.queue0.rx_bytes: 827196373649
                dev.igb.1.queue0.lro_queued: 0
                dev.igb.1.queue0.lro_flushed: 0
                dev.igb.1.mac_stats.excess_coll: 0
                dev.igb.1.mac_stats.single_coll: 0
                dev.igb.1.mac_stats.multiple_coll: 0
                dev.igb.1.mac_stats.late_coll: 0
                dev.igb.1.mac_stats.collision_count: 0
                dev.igb.1.mac_stats.symbol_errors: 0
                dev.igb.1.mac_stats.sequence_errors: 0
                dev.igb.1.mac_stats.defer_count: 0
                dev.igb.1.mac_stats.missed_packets: 0
                dev.igb.1.mac_stats.recv_no_buff: 0
                dev.igb.1.mac_stats.recv_undersize: 0
                dev.igb.1.mac_stats.recv_fragmented: 0
                dev.igb.1.mac_stats.recv_oversize: 0
                dev.igb.1.mac_stats.recv_jabber: 0
                dev.igb.1.mac_stats.recv_errs: 0
                dev.igb.1.mac_stats.crc_errs: 0
                dev.igb.1.mac_stats.alignment_errs: 0
                dev.igb.1.mac_stats.coll_ext_errs: 0
                dev.igb.1.mac_stats.xon_recvd: 0
                dev.igb.1.mac_stats.xon_txd: 0
                dev.igb.1.mac_stats.xoff_recvd: 0
                dev.igb.1.mac_stats.xoff_txd: 0
                dev.igb.1.mac_stats.total_pkts_recvd: 631666455
                dev.igb.1.mac_stats.good_pkts_recvd: 631652747
                dev.igb.1.mac_stats.bcast_pkts_recvd: 77781
                dev.igb.1.mac_stats.mcast_pkts_recvd: 145425
                dev.igb.1.mac_stats.rx_frames_64: 356
                dev.igb.1.mac_stats.rx_frames_65_127: 81351672
                dev.igb.1.mac_stats.rx_frames_128_255: 5213675
                dev.igb.1.mac_stats.rx_frames_256_511: 1833018
                dev.igb.1.mac_stats.rx_frames_512_1023: 1375058
                dev.igb.1.mac_stats.rx_frames_1024_1522: 541878968
                dev.igb.1.mac_stats.good_octets_recvd: 832249572479
                dev.igb.1.mac_stats.good_octets_txd: 1817163097562
                dev.igb.1.mac_stats.total_pkts_txd: 1749306409
                dev.igb.1.mac_stats.good_pkts_txd: 1749306409
                dev.igb.1.mac_stats.bcast_pkts_txd: 10921
                dev.igb.1.mac_stats.mcast_pkts_txd: 408712
                dev.igb.1.mac_stats.tx_frames_64: 7075834
                dev.igb.1.mac_stats.tx_frames_65_127: 524157938
                dev.igb.1.mac_stats.tx_frames_128_255: 17279963
                dev.igb.1.mac_stats.tx_frames_256_511: 10565548
                dev.igb.1.mac_stats.tx_frames_512_1023: 14796151
                dev.igb.1.mac_stats.tx_frames_1024_1522: 1175430975
                dev.igb.1.mac_stats.tso_txd: 0
                dev.igb.1.mac_stats.tso_ctx_fail: 0
                dev.igb.1.interrupts.asserts: 1372961031
                dev.igb.1.interrupts.rx_pkt_timer: 631643467
                dev.igb.1.interrupts.rx_abs_timer: 0
                dev.igb.1.interrupts.tx_pkt_timer: 0
                dev.igb.1.interrupts.tx_abs_timer: 631652747
                dev.igb.1.interrupts.tx_queue_empty: 1749288623
                dev.igb.1.interrupts.tx_queue_min_thresh: 0
                dev.igb.1.interrupts.rx_desc_min_thresh: 0
                dev.igb.1.interrupts.rx_overrun: 0
                dev.igb.1.host.breaker_tx_pkt: 0
                dev.igb.1.host.host_tx_pkt_discard: 0
                dev.igb.1.host.rx_pkt: 9280
                dev.igb.1.host.breaker_rx_pkts: 0
                dev.igb.1.host.breaker_rx_pkt_drop: 0
                dev.igb.1.host.tx_good_pkt: 17786
                dev.igb.1.host.breaker_tx_pkt_drop: 0
                dev.igb.1.host.rx_good_bytes: 832249624227
                dev.igb.1.host.tx_good_bytes: 1817163097562
                dev.igb.1.host.length_errors: 0
                dev.igb.1.host.serdes_violation_pkt: 0
                dev.igb.1.host.header_redir_missed: 0
                dev.igb.1.wake: 0
                dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
                dev.igb.2.%driver: igb
                dev.igb.2.%location: slot=0 function=1
                dev.igb.2.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
                dev.igb.2.%parent: pci11
                dev.igb.2.nvm: -1
                dev.igb.2.enable_aim: 1
                dev.igb.2.fc: 3
                dev.igb.2.rx_processing_limit: 100
                dev.igb.2.link_irq: 0
                dev.igb.2.dropped: 0
                dev.igb.2.tx_dma_fail: 0
                dev.igb.2.rx_overruns: 0
                dev.igb.2.watchdog_timeouts: 0
                dev.igb.2.device_control: 1087117889
                dev.igb.2.rx_control: 67141634
                dev.igb.2.interrupt_mask: 157
                dev.igb.2.extended_int_mask: 2147483648
                dev.igb.2.tx_buf_alloc: 0
                dev.igb.2.rx_buf_alloc: 0
                dev.igb.2.fc_high_water: 58976
                dev.igb.2.fc_low_water: 58960
                dev.igb.2.queue0.no_desc_avail: 0
                dev.igb.2.queue0.tx_packets: 476889
                dev.igb.2.queue0.rx_packets: 623282493
                dev.igb.2.queue0.rx_bytes: 74446064995
                dev.igb.2.queue0.lro_queued: 0
                dev.igb.2.queue0.lro_flushed: 0
                dev.igb.2.mac_stats.excess_coll: 0
                dev.igb.2.mac_stats.single_coll: 0
                dev.igb.2.mac_stats.multiple_coll: 0
                dev.igb.2.mac_stats.late_coll: 0
                dev.igb.2.mac_stats.collision_count: 0
                dev.igb.2.mac_stats.symbol_errors: 0
                dev.igb.2.mac_stats.sequence_errors: 0
                dev.igb.2.mac_stats.defer_count: 0
                dev.igb.2.mac_stats.missed_packets: 0
                dev.igb.2.mac_stats.recv_no_buff: 0
                dev.igb.2.mac_stats.recv_undersize: 0
                dev.igb.2.mac_stats.recv_fragmented: 0
                dev.igb.2.mac_stats.recv_oversize: 0
                dev.igb.2.mac_stats.recv_jabber: 0
                dev.igb.2.mac_stats.recv_errs: 0
                dev.igb.2.mac_stats.crc_errs: 0
                dev.igb.2.mac_stats.alignment_errs: 0
                dev.igb.2.mac_stats.coll_ext_errs: 0
                dev.igb.2.mac_stats.xon_recvd: 0
                dev.igb.2.mac_stats.xon_txd: 0
                dev.igb.2.mac_stats.xoff_recvd: 0
                dev.igb.2.mac_stats.xoff_txd: 0
                dev.igb.2.mac_stats.total_pkts_recvd: 697958370
                dev.igb.2.mac_stats.good_pkts_recvd: 623282231
                dev.igb.2.mac_stats.bcast_pkts_recvd: 129434
                dev.igb.2.mac_stats.mcast_pkts_recvd: 17096
                dev.igb.2.mac_stats.rx_frames_64: 1
                dev.igb.2.mac_stats.rx_frames_65_127: 563028186
                dev.igb.2.mac_stats.rx_frames_128_255: 28151786
                dev.igb.2.mac_stats.rx_frames_256_511: 9427339
                dev.igb.2.mac_stats.rx_frames_512_1023: 5069891
                dev.igb.2.mac_stats.rx_frames_1024_1522: 17605028
                dev.igb.2.mac_stats.good_octets_recvd: 79432278514
                dev.igb.2.mac_stats.good_octets_txd: 52903487
                dev.igb.2.mac_stats.total_pkts_txd: 476888
                dev.igb.2.mac_stats.good_pkts_txd: 476888
                dev.igb.2.mac_stats.bcast_pkts_txd: 11
                dev.igb.2.mac_stats.mcast_pkts_txd: 165656
                dev.igb.2.mac_stats.tx_frames_64: 7
                dev.igb.2.mac_stats.tx_frames_65_127: 300650
                dev.igb.2.mac_stats.tx_frames_128_255: 172846
                dev.igb.2.mac_stats.tx_frames_256_511: 3385
                dev.igb.2.mac_stats.tx_frames_512_1023: 0
                dev.igb.2.mac_stats.tx_frames_1024_1522: 0
                dev.igb.2.mac_stats.tso_txd: 0
                dev.igb.2.mac_stats.tso_ctx_fail: 0
                dev.igb.2.interrupts.asserts: 591617753
                dev.igb.2.interrupts.rx_pkt_timer: 623273921
                dev.igb.2.interrupts.rx_abs_timer: 0
                dev.igb.2.interrupts.tx_pkt_timer: 0
                dev.igb.2.interrupts.tx_abs_timer: 623282231
                dev.igb.2.interrupts.tx_queue_empty: 476827
                dev.igb.2.interrupts.tx_queue_min_thresh: 0
                dev.igb.2.interrupts.rx_desc_min_thresh: 0
                dev.igb.2.interrupts.rx_overrun: 0
                dev.igb.2.host.breaker_tx_pkt: 0
                dev.igb.2.host.host_tx_pkt_discard: 0
                dev.igb.2.host.rx_pkt: 8310
                dev.igb.2.host.breaker_rx_pkts: 0
                dev.igb.2.host.breaker_rx_pkt_drop: 0
                dev.igb.2.host.tx_good_pkt: 61
                dev.igb.2.host.breaker_tx_pkt_drop: 0
                dev.igb.2.host.rx_good_bytes: 79432278736
                dev.igb.2.host.tx_good_bytes: 52903487
                dev.igb.2.host.length_errors: 0
                dev.igb.2.host.serdes_violation_pkt: 0
                dev.igb.2.host.header_redir_missed: 0
                
                
                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.