Errors all over the place
-
Hello.
Originally, my setup was functioning great, it was a standalone box. But then I went and merged it with FreeNAS box using ESXi 6.0 and hardware passthrough for my NICs and HBA. Everything seems working fine for about a month now, but I started getting IN errors on WAN interface and they continue incrementing..LAN is a LACP LAGG of 2 ports on fiber igb card connected to switch, which breaks up into serveral VLANs. Works fine exept very few OUT errors caused by LACP itself when systems are up but OSes are not, eg. in BIOS, when LACP is not initialized yet.
WAN is a simple Intel Desktop CT adapter, so em driver.
In front there is an ONT modem, which "does not support bridge mode" accroding to my ISP, so WAN space is another LAN..All offloading options are disabled.
Here is interface status:
Same count in "dev.em.0.mac_stats.missed_packets: 4023674":
$ sysctl dev.em dev.em.%parent: dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.4.2 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE60.S1F0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 dev.em.0.%parent: pci19 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.fc: 3 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.itr: 488 dev.em.0.rx_processing_limit: 100 dev.em.0.eee_control: 1 dev.em.0.link_irq: 0 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.rx_overruns: 196259 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1477444168 dev.em.0.rx_control: 67141634 dev.em.0.fc_high_water: 18432 dev.em.0.fc_low_water: 16932 dev.em.0.queue0.txd_head: 793 dev.em.0.queue0.txd_tail: 793 dev.em.0.queue0.tx_irq: 0 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 624 dev.em.0.queue0.rxd_tail: 623 dev.em.0.queue0.rx_irq: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 4023674 dev.em.0.mac_stats.recv_no_buff: 1147 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 229889 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 4253562 dev.em.0.mac_stats.total_pkts_recvd: 1019190241 dev.em.0.mac_stats.good_pkts_recvd: 1015166567 dev.em.0.mac_stats.bcast_pkts_recvd: 43766 dev.em.0.mac_stats.mcast_pkts_recvd: 37032 dev.em.0.mac_stats.rx_frames_64: 37869404 dev.em.0.mac_stats.rx_frames_65_127: 46904834 dev.em.0.mac_stats.rx_frames_128_255: 70985248 dev.em.0.mac_stats.rx_frames_256_511: 29553892 dev.em.0.mac_stats.rx_frames_512_1023: 74673383 dev.em.0.mac_stats.rx_frames_1024_1522: 755179806 dev.em.0.mac_stats.good_octets_recvd: 1184386034957 dev.em.0.mac_stats.good_octets_txd: 150188980069 dev.em.0.mac_stats.total_pkts_txd: 811219743 dev.em.0.mac_stats.good_pkts_txd: 806736292 dev.em.0.mac_stats.bcast_pkts_txd: 120 dev.em.0.mac_stats.mcast_pkts_txd: 11 dev.em.0.mac_stats.tx_frames_64: 161896304 dev.em.0.mac_stats.tx_frames_65_127: 507949485 dev.em.0.mac_stats.tx_frames_128_255: 59335118 dev.em.0.mac_stats.tx_frames_256_511: 15082844 dev.em.0.mac_stats.tx_frames_512_1023: 4651980 dev.em.0.mac_stats.tx_frames_1024_1522: 57820561 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 832965679 dev.em.0.interrupts.rx_pkt_timer: 73507 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 31105 dev.em.0.interrupts.tx_abs_timer: 40799 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 77 dev.em.0.wake: 0
I've swapped cable with no luck. NIC itself I believe is fine too since it all started after moving into virtual enviromnent. I have 2 more NICs alike to test, but would like to avoid downtime needed for hadware swap if possible.
So what are these 'missed packets'? And should I be worried?
Googling gave no answers. -
During last 8 hours there were about 40k new errors under load (79 Mb/s on 95th percentile)… Anyone?
-
dev.em.0.mac_stats.missed_packets: 4023674 dev.em.0.mac_stats.recv_no_buff: 1147
Looks like the packets are coming in too fast for the NIC/driver/OS to handle. What version of pfSense? Are you sure the hardware is not being overloaded?
-
I highly doubt it. I mean, I'm far from being good at networking specifics, and may only guess about reasons of such issue as mine, so I would rather provide more experienced people with as much raw data as I can. At least it feels like the most efficient path I can take in that situation. I appologise for that pile of information in advance, but setup is somewhat sophisticated.
Exact specs for ESXi host running only pfSense and FreeNAS are:
►CPU: Intel Xeon E3-1220V3
►MB: Supermicro X10SLM-F
►RAM: 32 GB KVR16E11/8
►HDD: HDS721050CLA362
►Box: Fractal Design Define R4 Black
►PSU: Chieftec GPS-700A8 @ 100W
►UPS: APC Back-UPS BC650-RS
pfSense passthrough:
►WAN: Intel EXPI9301CT
►Trunk: Intel E1G42EFBLK
FreeNAS passthrough & spinning rust:
►HBA: IBM ServeRAID M1115 [LSI9211-8i fw.20 IT w/o bios]
►'RAID1': 3x 2TB [WD20EARS + WD20EZRX + WD2001FASS]
►'JBOD': 3x 4TB HDS5C4040ALE630 + 4TB WD40EZRXpfSense packages: squid, Lightsquid, Service Watchdog, nut, OpenVPN Client Export Utility, Open-VM-Tools + frequent use of OpenVPN RA Server for traffic redirection and data sync
FreeNAS: Syncthing live backups form Windows machines across the network, couple Plex jails, NFS/SMB shared database and data storage for Proxmox node (OwnCloud, p2p, etc), lazy compression, snapshoting, none of hard stuff like ZFS deduplication or encryption..pfSense & FreeNAS are on latest stable release versions, ESXi is at 6.0.0, 3029758
And it's a home host serving only as high load as members of average sized family can produce..
CPU is idle most of the time (experiences full frequency loads probably only at night while generating video preview thumbnails for Plex's new media. And even for 100% utilization scenario I made CPU reservation of 1000Mhz for pfSense VM).
RAM: 4GB pfSense (50% allowed for & often used by squid) / 23GB FreeNAS / rest reserved by ESXi.
MBUF Usage is most of the time at 20%, not sure whether it can spike to a serious amount, never noticed..
WAN throughput from ISP is 100 mbit bidirectional with 200 mbit Rx burst.
Typical load - judge yourself: RRD graphsBy my expectations, that host should be more than capable of my tasks if not overkill (except in RAM for ZFS, it's on border line, but I don't plan to expand on that particular platform). What's more important, I consider that system built using all crucial rules for both OSes, although not running on bare metal.
Correct me if I'm wrong. -
New details.
While monitoring the system today I was looking at the interface statistics, wondering what makes diference between WAN and other interfaces. Because at least two of them experience equal or higher load than WAN. Then I realized that I'm not seeing the whole picture because other stats are for virtual (VLAN) interfaces rather than physical as em0 for WAN. So pfSense accounts for errors to send out packets from these interfaces but can not divide actual error data on physical NIC for each virtual interface..$ sysctl dev.igb
dev.igb.%parent: dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000 dev.igb.0.%parent: pci3 dev.igb.0.nvm: -1 dev.igb.0.enable_aim: 1 dev.igb.0.fc: 3 dev.igb.0.rx_processing_limit: 100 dev.igb.0.link_irq: 0 dev.igb.0.dropped: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.rx_overruns: 404 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.device_control: 1087117889 dev.igb.0.rx_control: 67141634 dev.igb.0.interrupt_mask: 157 dev.igb.0.extended_int_mask: 2147483648 dev.igb.0.tx_buf_alloc: 0 dev.igb.0.rx_buf_alloc: 0 dev.igb.0.fc_high_water: 58976 dev.igb.0.fc_low_water: 58960 dev.igb.0.queue0.no_desc_avail: 1261891 dev.igb.0.queue0.tx_packets: 1313009825 dev.igb.0.queue0.rx_packets: 781759341 dev.igb.0.queue0.rx_bytes: 580723340999 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.missed_packets: 83631 dev.igb.0.mac_stats.recv_no_buff: 25953 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.total_pkts_recvd: 782461121 dev.igb.0.mac_stats.good_pkts_recvd: 781759301 dev.igb.0.mac_stats.bcast_pkts_recvd: 1520864 dev.igb.0.mac_stats.mcast_pkts_recvd: 203559 dev.igb.0.mac_stats.rx_frames_64: 81560000 dev.igb.0.mac_stats.rx_frames_65_127: 277073969 dev.igb.0.mac_stats.rx_frames_128_255: 43744716 dev.igb.0.mac_stats.rx_frames_256_511: 15270084 dev.igb.0.mac_stats.rx_frames_512_1023: 4240271 dev.igb.0.mac_stats.rx_frames_1024_1522: 359870261 dev.igb.0.mac_stats.good_octets_recvd: 586977128523 dev.igb.0.mac_stats.good_octets_txd: 1351374080382 dev.igb.0.mac_stats.total_pkts_txd: 1313009811 dev.igb.0.mac_stats.good_pkts_txd: 1313009811 dev.igb.0.mac_stats.bcast_pkts_txd: 30114 dev.igb.0.mac_stats.mcast_pkts_txd: 1609000 dev.igb.0.mac_stats.tx_frames_64: 16842087 dev.igb.0.mac_stats.tx_frames_65_127: 336263799 dev.igb.0.mac_stats.tx_frames_128_255: 41689908 dev.igb.0.mac_stats.tx_frames_256_511: 18165668 dev.igb.0.mac_stats.tx_frames_512_1023: 36777014 dev.igb.0.mac_stats.tx_frames_1024_1522: 863271335 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.interrupts.asserts: 1458531356 dev.igb.0.interrupts.rx_pkt_timer: 781750145 dev.igb.0.interrupts.rx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 0 dev.igb.0.interrupts.tx_abs_timer: 781754685 dev.igb.0.interrupts.tx_queue_empty: 1312992017 dev.igb.0.interrupts.tx_queue_min_thresh: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.rx_pkt: 9156 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 17794 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.rx_good_bytes: 586977148059 dev.igb.0.host.tx_good_bytes: 1351374080382 dev.igb.0.host.length_errors: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.header_redir_missed: 0 dev.igb.0.wake: 0 dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000 dev.igb.1.%parent: pci3 dev.igb.1.nvm: -1 dev.igb.1.enable_aim: 1 dev.igb.1.fc: 3 dev.igb.1.rx_processing_limit: 100 dev.igb.1.link_irq: 0 dev.igb.1.dropped: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.rx_overruns: 7 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.device_control: 1087117889 dev.igb.1.rx_control: 67141634 dev.igb.1.interrupt_mask: 157 dev.igb.1.extended_int_mask: 2147483648 dev.igb.1.tx_buf_alloc: 0 dev.igb.1.rx_buf_alloc: 0 dev.igb.1.fc_high_water: 58976 dev.igb.1.fc_low_water: 58960 dev.igb.1.queue0.no_desc_avail: 26818 dev.igb.1.queue0.tx_packets: 1312415919 dev.igb.1.queue0.rx_packets: 784960625 dev.igb.1.queue0.rx_bytes: 657958891610 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.missed_packets: 1074 dev.igb.1.mac_stats.recv_no_buff: 280 dev.igb.1.mac_stats.recv_undersize: 0 dev.igb.1.mac_stats.recv_fragmented: 0 dev.igb.1.mac_stats.recv_oversize: 0 dev.igb.1.mac_stats.recv_jabber: 0 dev.igb.1.mac_stats.recv_errs: 0 dev.igb.1.mac_stats.crc_errs: 0 dev.igb.1.mac_stats.alignment_errs: 0 dev.igb.1.mac_stats.coll_ext_errs: 0 dev.igb.1.mac_stats.xon_recvd: 0 dev.igb.1.mac_stats.xon_txd: 0 dev.igb.1.mac_stats.xoff_recvd: 0 dev.igb.1.mac_stats.xoff_txd: 0 dev.igb.1.mac_stats.total_pkts_recvd: 1168917353 dev.igb.1.mac_stats.good_pkts_recvd: 784960599 dev.igb.1.mac_stats.bcast_pkts_recvd: 405858 dev.igb.1.mac_stats.mcast_pkts_recvd: 294354 dev.igb.1.mac_stats.rx_frames_64: 89494759 dev.igb.1.mac_stats.rx_frames_65_127: 232942520 dev.igb.1.mac_stats.rx_frames_128_255: 35599853 dev.igb.1.mac_stats.rx_frames_256_511: 8289293 dev.igb.1.mac_stats.rx_frames_512_1023: 3432957 dev.igb.1.mac_stats.rx_frames_1024_1522: 415201217 dev.igb.1.mac_stats.good_octets_recvd: 664238288288 dev.igb.1.mac_stats.good_octets_txd: 1161071633488 dev.igb.1.mac_stats.total_pkts_txd: 1312415798 dev.igb.1.mac_stats.good_pkts_txd: 1312415798 dev.igb.1.mac_stats.bcast_pkts_txd: 197034 dev.igb.1.mac_stats.mcast_pkts_txd: 1608910 dev.igb.1.mac_stats.tx_frames_64: 263810305 dev.igb.1.mac_stats.tx_frames_65_127: 201290334 dev.igb.1.mac_stats.tx_frames_128_255: 57384858 dev.igb.1.mac_stats.tx_frames_256_511: 21661731 dev.igb.1.mac_stats.tx_frames_512_1023: 47026530 dev.igb.1.mac_stats.tx_frames_1024_1522: 721242040 dev.igb.1.mac_stats.tso_txd: 0 dev.igb.1.mac_stats.tso_ctx_fail: 0 dev.igb.1.interrupts.asserts: 1354951558 dev.igb.1.interrupts.rx_pkt_timer: 784951616 dev.igb.1.interrupts.rx_abs_timer: 0 dev.igb.1.interrupts.tx_pkt_timer: 0 dev.igb.1.interrupts.tx_abs_timer: 784960591 dev.igb.1.interrupts.tx_queue_empty: 1312400215 dev.igb.1.interrupts.tx_queue_min_thresh: 0 dev.igb.1.interrupts.rx_desc_min_thresh: 0 dev.igb.1.interrupts.rx_overrun: 0 dev.igb.1.host.breaker_tx_pkt: 0 dev.igb.1.host.host_tx_pkt_discard: 0 dev.igb.1.host.rx_pkt: 8983 dev.igb.1.host.breaker_rx_pkts: 0 dev.igb.1.host.breaker_rx_pkt_drop: 0 dev.igb.1.host.tx_good_pkt: 15583 dev.igb.1.host.breaker_tx_pkt_drop: 0 dev.igb.1.host.rx_good_bytes: 664238306626 dev.igb.1.host.tx_good_bytes: 1161071633488 dev.igb.1.host.length_errors: 0 dev.igb.1.host.serdes_violation_pkt: 0 dev.igb.1.host.header_redir_missed: 0
dev.igb.0.mac_stats.missed_packets: 83631
dev.igb.0.mac_stats.recv_no_buff: 25953
dev.igb.1.mac_stats.missed_packets: 1074
dev.igb.1.mac_stats.recv_no_buff: 280So the problem is system wide :( or not really since VMXNET3 adapter is pretty clean?
Also, using simple math, there was about 340k new errors on WAN during last 23 hours. That's ~15k per hour, which is 3 times more than during 8 hours period. Load during 23h was approx 17Mb/s vs 79Mb/s during 8h (95th percentile according to RRD).
So issue should hardly be dependent on load, right?
-
Well, I made a progress on incoming errors..
These errors accumulated rapidly and while doing no observable harm. But about a week ago situation became worse. On a month of uptime system freaked out, error counter went through the roof and WAN went down. I rebooted.. And it repeated 3 times during 1st or 2nd 24 hours of uptime since then (under load). And that's when I started playing with hardware and poking drivers with a stick. So, the results:-
EXPI9301CT NIC itself is working fine, I tried swapping with the same exact model - same problems. Missed packets and "recv_no_buff" counters. Even on igb card, while this one is installed. So it should be driver (em) or this specific model.
-
Changing model, at first I used I217-LM (one of onboard NICs, em driver) for WAN passthrough. According to vSphere Client it has own pci lane, unlike other one, which seems to be interconnected through PCH. "That one should be superior," - I thought - "as long as they are pretty similar looking on Intel ARK". As a result, there was no incrementing error counters from the start on any interface, but during 2nd 24 hours of "stressing" the system with 80 mbps on average, WAN timed out for couple seconds (I actually experienced it), interface dropped 923 errors and continued functioning. Flimsy.. Although no errors on dev.igb. So, again should be driver (em) or this specific model (if momentary glitch was unrelated)
-
Then I switched WAN to another onboard NIC - I210 (igb driver). No incoming errors, no missed packets. Same on fiber card. I pushed roughly the same amount of traffic through system in 2 days which initially brought my WAN down after the month of runtime. System still stable. Missed packets and "recv_no_buff" at 0 on all NICs.
To sum up, I still don't know whether it's the specific model that causes issue during passthrough (while it is actually functional and now acts as a secondary LACP port for ESXi management in place of onboard one that is now passed to pfSense for WAN) or the em driver and there could be other em NICs having same issue with passthrough or even the 82574L chip has something to do in hardware. EXPI9301CT is a very old model meant for desktops, not virtual environments, although it is still present in Intel's 2016 catalogue. And also given that em dirver in FreeBSD man pages is dated October 11, 2011 while igb is of March 25, 2013, I just made a conclusion for myself that later is just more stable and Intel NICs are not bulletproof as I thought.
I don't know what are these outgoing errors.. I recall having them all the time, even on bare metal, even on other platform. There is literally no info I managed to find about any possible causes of them or at least the meaning of these counters, so if someone may enlighten me, please do.. At least, they seem to not be as harmless.
$ sysctl dev.igb
dev.igb.%parent: dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 class=0x020000 dev.igb.0.%parent: pci3 dev.igb.0.nvm: -1 dev.igb.0.enable_aim: 1 dev.igb.0.fc: 3 dev.igb.0.rx_processing_limit: 100 dev.igb.0.dmac: 0 dev.igb.0.eee_disabled: 0 dev.igb.0.link_irq: 0 dev.igb.0.dropped: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.rx_overruns: 0 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.device_control: 1478230593 dev.igb.0.rx_control: 71335938 dev.igb.0.interrupt_mask: 157 dev.igb.0.extended_int_mask: 2147483648 dev.igb.0.tx_buf_alloc: 0 dev.igb.0.rx_buf_alloc: 0 dev.igb.0.fc_high_water: 31328 dev.igb.0.fc_low_water: 31312 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.tx_packets: 620869912 dev.igb.0.queue0.rx_packets: 618707893 dev.igb.0.queue0.rx_bytes: 851127332692 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.missed_packets: 0 dev.igb.0.mac_stats.recv_no_buff: 0 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.total_pkts_recvd: 618715218 dev.igb.0.mac_stats.good_pkts_recvd: 618707618 dev.igb.0.mac_stats.bcast_pkts_recvd: 1428 dev.igb.0.mac_stats.mcast_pkts_recvd: 3090 dev.igb.0.mac_stats.rx_frames_64: 601133 dev.igb.0.mac_stats.rx_frames_65_127: 13638243 dev.igb.0.mac_stats.rx_frames_128_255: 10436929 dev.igb.0.mac_stats.rx_frames_256_511: 9534739 dev.igb.0.mac_stats.rx_frames_512_1023: 13436838 dev.igb.0.mac_stats.rx_frames_1024_1522: 571059736 dev.igb.0.mac_stats.good_octets_recvd: 853601812367 dev.igb.0.mac_stats.good_octets_txd: 71259681700 dev.igb.0.mac_stats.total_pkts_txd: 620869651 dev.igb.0.mac_stats.good_pkts_txd: 620869651 dev.igb.0.mac_stats.bcast_pkts_txd: 3 dev.igb.0.mac_stats.mcast_pkts_txd: 3 dev.igb.0.mac_stats.tx_frames_64: 3612420 dev.igb.0.mac_stats.tx_frames_65_127: 560906135 dev.igb.0.mac_stats.tx_frames_128_255: 28742303 dev.igb.0.mac_stats.tx_frames_256_511: 8907879 dev.igb.0.mac_stats.tx_frames_512_1023: 4722579 dev.igb.0.mac_stats.tx_frames_1024_1522: 13978335 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.interrupts.asserts: 965490770 dev.igb.0.interrupts.rx_pkt_timer: 618687254 dev.igb.0.interrupts.rx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 0 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_queue_empty: 620853924 dev.igb.0.interrupts.tx_queue_min_thresh: 618707618 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.rx_pkt: 20364 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 15727 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.rx_good_bytes: 853601838827 dev.igb.0.host.tx_good_bytes: 71259681700 dev.igb.0.host.length_errors: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.header_redir_missed: 0 dev.igb.0.wake: 0 dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=0 handle=\_SB_.PCI0.PE50.S1F0 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000 dev.igb.1.%parent: pci11 dev.igb.1.nvm: -1 dev.igb.1.enable_aim: 1 dev.igb.1.fc: 3 dev.igb.1.rx_processing_limit: 100 dev.igb.1.link_irq: 0 dev.igb.1.dropped: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.rx_overruns: 0 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.device_control: 1087117889 dev.igb.1.rx_control: 67141634 dev.igb.1.interrupt_mask: 157 dev.igb.1.extended_int_mask: 2147483648 dev.igb.1.tx_buf_alloc: 0 dev.igb.1.rx_buf_alloc: 0 dev.igb.1.fc_high_water: 58976 dev.igb.1.fc_low_water: 58960 dev.igb.1.queue0.no_desc_avail: 3 dev.igb.1.queue0.tx_packets: 1749306695 dev.igb.1.queue0.rx_packets: 631652753 dev.igb.1.queue0.rx_bytes: 827196373649 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.missed_packets: 0 dev.igb.1.mac_stats.recv_no_buff: 0 dev.igb.1.mac_stats.recv_undersize: 0 dev.igb.1.mac_stats.recv_fragmented: 0 dev.igb.1.mac_stats.recv_oversize: 0 dev.igb.1.mac_stats.recv_jabber: 0 dev.igb.1.mac_stats.recv_errs: 0 dev.igb.1.mac_stats.crc_errs: 0 dev.igb.1.mac_stats.alignment_errs: 0 dev.igb.1.mac_stats.coll_ext_errs: 0 dev.igb.1.mac_stats.xon_recvd: 0 dev.igb.1.mac_stats.xon_txd: 0 dev.igb.1.mac_stats.xoff_recvd: 0 dev.igb.1.mac_stats.xoff_txd: 0 dev.igb.1.mac_stats.total_pkts_recvd: 631666455 dev.igb.1.mac_stats.good_pkts_recvd: 631652747 dev.igb.1.mac_stats.bcast_pkts_recvd: 77781 dev.igb.1.mac_stats.mcast_pkts_recvd: 145425 dev.igb.1.mac_stats.rx_frames_64: 356 dev.igb.1.mac_stats.rx_frames_65_127: 81351672 dev.igb.1.mac_stats.rx_frames_128_255: 5213675 dev.igb.1.mac_stats.rx_frames_256_511: 1833018 dev.igb.1.mac_stats.rx_frames_512_1023: 1375058 dev.igb.1.mac_stats.rx_frames_1024_1522: 541878968 dev.igb.1.mac_stats.good_octets_recvd: 832249572479 dev.igb.1.mac_stats.good_octets_txd: 1817163097562 dev.igb.1.mac_stats.total_pkts_txd: 1749306409 dev.igb.1.mac_stats.good_pkts_txd: 1749306409 dev.igb.1.mac_stats.bcast_pkts_txd: 10921 dev.igb.1.mac_stats.mcast_pkts_txd: 408712 dev.igb.1.mac_stats.tx_frames_64: 7075834 dev.igb.1.mac_stats.tx_frames_65_127: 524157938 dev.igb.1.mac_stats.tx_frames_128_255: 17279963 dev.igb.1.mac_stats.tx_frames_256_511: 10565548 dev.igb.1.mac_stats.tx_frames_512_1023: 14796151 dev.igb.1.mac_stats.tx_frames_1024_1522: 1175430975 dev.igb.1.mac_stats.tso_txd: 0 dev.igb.1.mac_stats.tso_ctx_fail: 0 dev.igb.1.interrupts.asserts: 1372961031 dev.igb.1.interrupts.rx_pkt_timer: 631643467 dev.igb.1.interrupts.rx_abs_timer: 0 dev.igb.1.interrupts.tx_pkt_timer: 0 dev.igb.1.interrupts.tx_abs_timer: 631652747 dev.igb.1.interrupts.tx_queue_empty: 1749288623 dev.igb.1.interrupts.tx_queue_min_thresh: 0 dev.igb.1.interrupts.rx_desc_min_thresh: 0 dev.igb.1.interrupts.rx_overrun: 0 dev.igb.1.host.breaker_tx_pkt: 0 dev.igb.1.host.host_tx_pkt_discard: 0 dev.igb.1.host.rx_pkt: 9280 dev.igb.1.host.breaker_rx_pkts: 0 dev.igb.1.host.breaker_rx_pkt_drop: 0 dev.igb.1.host.tx_good_pkt: 17786 dev.igb.1.host.breaker_tx_pkt_drop: 0 dev.igb.1.host.rx_good_bytes: 832249624227 dev.igb.1.host.tx_good_bytes: 1817163097562 dev.igb.1.host.length_errors: 0 dev.igb.1.host.serdes_violation_pkt: 0 dev.igb.1.host.header_redir_missed: 0 dev.igb.1.wake: 0 dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0 dev.igb.2.%driver: igb dev.igb.2.%location: slot=0 function=1 dev.igb.2.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000 dev.igb.2.%parent: pci11 dev.igb.2.nvm: -1 dev.igb.2.enable_aim: 1 dev.igb.2.fc: 3 dev.igb.2.rx_processing_limit: 100 dev.igb.2.link_irq: 0 dev.igb.2.dropped: 0 dev.igb.2.tx_dma_fail: 0 dev.igb.2.rx_overruns: 0 dev.igb.2.watchdog_timeouts: 0 dev.igb.2.device_control: 1087117889 dev.igb.2.rx_control: 67141634 dev.igb.2.interrupt_mask: 157 dev.igb.2.extended_int_mask: 2147483648 dev.igb.2.tx_buf_alloc: 0 dev.igb.2.rx_buf_alloc: 0 dev.igb.2.fc_high_water: 58976 dev.igb.2.fc_low_water: 58960 dev.igb.2.queue0.no_desc_avail: 0 dev.igb.2.queue0.tx_packets: 476889 dev.igb.2.queue0.rx_packets: 623282493 dev.igb.2.queue0.rx_bytes: 74446064995 dev.igb.2.queue0.lro_queued: 0 dev.igb.2.queue0.lro_flushed: 0 dev.igb.2.mac_stats.excess_coll: 0 dev.igb.2.mac_stats.single_coll: 0 dev.igb.2.mac_stats.multiple_coll: 0 dev.igb.2.mac_stats.late_coll: 0 dev.igb.2.mac_stats.collision_count: 0 dev.igb.2.mac_stats.symbol_errors: 0 dev.igb.2.mac_stats.sequence_errors: 0 dev.igb.2.mac_stats.defer_count: 0 dev.igb.2.mac_stats.missed_packets: 0 dev.igb.2.mac_stats.recv_no_buff: 0 dev.igb.2.mac_stats.recv_undersize: 0 dev.igb.2.mac_stats.recv_fragmented: 0 dev.igb.2.mac_stats.recv_oversize: 0 dev.igb.2.mac_stats.recv_jabber: 0 dev.igb.2.mac_stats.recv_errs: 0 dev.igb.2.mac_stats.crc_errs: 0 dev.igb.2.mac_stats.alignment_errs: 0 dev.igb.2.mac_stats.coll_ext_errs: 0 dev.igb.2.mac_stats.xon_recvd: 0 dev.igb.2.mac_stats.xon_txd: 0 dev.igb.2.mac_stats.xoff_recvd: 0 dev.igb.2.mac_stats.xoff_txd: 0 dev.igb.2.mac_stats.total_pkts_recvd: 697958370 dev.igb.2.mac_stats.good_pkts_recvd: 623282231 dev.igb.2.mac_stats.bcast_pkts_recvd: 129434 dev.igb.2.mac_stats.mcast_pkts_recvd: 17096 dev.igb.2.mac_stats.rx_frames_64: 1 dev.igb.2.mac_stats.rx_frames_65_127: 563028186 dev.igb.2.mac_stats.rx_frames_128_255: 28151786 dev.igb.2.mac_stats.rx_frames_256_511: 9427339 dev.igb.2.mac_stats.rx_frames_512_1023: 5069891 dev.igb.2.mac_stats.rx_frames_1024_1522: 17605028 dev.igb.2.mac_stats.good_octets_recvd: 79432278514 dev.igb.2.mac_stats.good_octets_txd: 52903487 dev.igb.2.mac_stats.total_pkts_txd: 476888 dev.igb.2.mac_stats.good_pkts_txd: 476888 dev.igb.2.mac_stats.bcast_pkts_txd: 11 dev.igb.2.mac_stats.mcast_pkts_txd: 165656 dev.igb.2.mac_stats.tx_frames_64: 7 dev.igb.2.mac_stats.tx_frames_65_127: 300650 dev.igb.2.mac_stats.tx_frames_128_255: 172846 dev.igb.2.mac_stats.tx_frames_256_511: 3385 dev.igb.2.mac_stats.tx_frames_512_1023: 0 dev.igb.2.mac_stats.tx_frames_1024_1522: 0 dev.igb.2.mac_stats.tso_txd: 0 dev.igb.2.mac_stats.tso_ctx_fail: 0 dev.igb.2.interrupts.asserts: 591617753 dev.igb.2.interrupts.rx_pkt_timer: 623273921 dev.igb.2.interrupts.rx_abs_timer: 0 dev.igb.2.interrupts.tx_pkt_timer: 0 dev.igb.2.interrupts.tx_abs_timer: 623282231 dev.igb.2.interrupts.tx_queue_empty: 476827 dev.igb.2.interrupts.tx_queue_min_thresh: 0 dev.igb.2.interrupts.rx_desc_min_thresh: 0 dev.igb.2.interrupts.rx_overrun: 0 dev.igb.2.host.breaker_tx_pkt: 0 dev.igb.2.host.host_tx_pkt_discard: 0 dev.igb.2.host.rx_pkt: 8310 dev.igb.2.host.breaker_rx_pkts: 0 dev.igb.2.host.breaker_rx_pkt_drop: 0 dev.igb.2.host.tx_good_pkt: 61 dev.igb.2.host.breaker_tx_pkt_drop: 0 dev.igb.2.host.rx_good_bytes: 79432278736 dev.igb.2.host.tx_good_bytes: 52903487 dev.igb.2.host.length_errors: 0 dev.igb.2.host.serdes_violation_pkt: 0 dev.igb.2.host.header_redir_missed: 0
-