NIC only works with polling enabled
-
Got a bit of a weird problem, it seems I have a NIC that only works (sort of) when I enable polling. The hardware is the OPNsense 5 port GHz rack edition from applianceshop.eu. Unfortunately the website is down at the moment. EDIT: website is back up now
It has 5 NICs in total with 3 different types: 1 x re, 2 x rl & 2 x em. The re & rl ones work fine with polling off or on. The em ones only seem to work with polling on but this makes the Webconfigurator and SSH access really slow. Additionally the em nics seem to average 20ms latency on the LAN. I found I can turn on polling just for the em NIC by running "ifconfig em0 polling" which solves the slow Webconfigurator and SSH but this turns back off every time I make a change to that NIC or the system reboots.
Unfortunately I don't know if this box exhibits the same behavior on v1.2.3 as we have only recently needed to use the em based interfaces and we upgraded the box to 2.0 rc1 i386 ages ago (which has been running flawlessly). I updated it to 2.0RC2 yesterday but the behavior is the same.
Does anyone have any further troubleshooting steps I can try? Or as a work around, how could I get polling to stay on just for the em NICs? I have been using pfsense for a couple of years now but since I have had very few problems I have no idea how to troubleshoot this particular problem!
Thanks in advance,
Paul
Some additional info:
dmesg | grep em em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3=""> port 0xc880-0xc8bf mem 0xcfce0000-0xcfcfffff,0xcfcc0000-0xcfcdffff irq 11 at device 1.0 on pci2 em0: [FILTER] em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3=""> port 0xc480-0xc4bf mem 0xcfc60000-0xcfc7ffff,0xcfc40000-0xcfc5ffff irq 16 at device 4.0 on pci2 em1: [FILTER]</intel(r)></intel(r)>
ifconfig em0 em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=20db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_magic>ether 00:03:1d:09:39:84 inet6 fe80::203:1dff:fe09:3984%em0 prefixlen 64 scopeid 0x1 inet xx.xx.xx.xx netmask 0xfffffff0 broadcast xx.xx.xx.xx nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>) status: active</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>
-
Hmm, weird problem. What are the NICs on the em interfaces?
You could add the polling work around as a shell command. That would only help with rebooting though.
Steve
-
Is there any way to identify the make/model of the NIC from the command line? This box is at a remote site so I can't look inside.
I'll give that shell command ago.
-
[2.0-RC2][root@pfsense.fire.box]/root(4): sysctl dev.em.0 dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 dev.em.0.%driver: em dev.em.0.%location: slot=1 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1075 subvendor=0x8086 subdevice=0x1075 class=0x020000
Plus a load of other variables. Change 0 for 1 for the other NIC, though they're probably the same.
Steve
-
Thanks, both are identical, output below:
dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 dev.em.0.%driver: em dev.em.0.%location: slot=1 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1076 subvendor=0x8086 subdevice=0x1076 class=0x020000 dev.em.0.%parent: pci2 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1480327745 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 47104 dev.em.0.fc_low_water: 45604 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 209 dev.em.0.txd_tail: 209 dev.em.0.rxd_head: 152 dev.em.0.rxd_tail: 151 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 1546 dev.em.0.mac_stats.recv_no_buff: 302 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 2 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 1548 dev.em.0.mac_stats.total_pkts_recvd: 1155933 dev.em.0.mac_stats.good_pkts_recvd: 1154387 dev.em.0.mac_stats.bcast_pkts_recvd: 827665 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 584176 dev.em.0.mac_stats.rx_frames_65_127: 257893 dev.em.0.mac_stats.rx_frames_128_255: 109244 dev.em.0.mac_stats.rx_frames_256_511: 3437 dev.em.0.mac_stats.rx_frames_512_1023: 1107 dev.em.0.mac_stats.rx_frames_1024_1522: 198530 dev.em.0.mac_stats.good_octets_recvd: 384175227 dev.em.0.mac_stats.good_octets_txd: 93827474 dev.em.0.mac_stats.total_pkts_txd: 422787 dev.em.0.mac_stats.good_pkts_txd: 421237 dev.em.0.mac_stats.bcast_pkts_txd: 1812 dev.em.0.mac_stats.mcast_pkts_txd: 3 dev.em.0.mac_stats.tx_frames_64: 1364 dev.em.0.mac_stats.tx_frames_65_127: 178187 dev.em.0.mac_stats.tx_frames_128_255: 196500 dev.em.0.mac_stats.tx_frames_256_511: 9443 dev.em.0.mac_stats.tx_frames_512_1023: 10213 dev.em.0.mac_stats.tx_frames_1024_1522: 25530 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0
-
Is that with polling enabled?
Steve
-
Yeah it was, output looks like this with it disabled:
dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 dev.em.0.%driver: em dev.em.0.%location: slot=1 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1076 subvendor=0x8086 subdevice=0x1076 class=0x020000 dev.em.0.%parent: pci2 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1480327745 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 47104 dev.em.0.fc_low_water: 45604 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 122 dev.em.0.txd_tail: 122 dev.em.0.rxd_head: 235 dev.em.0.rxd_tail: 235 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 1546 dev.em.0.mac_stats.recv_no_buff: 403 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 2 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 1548 dev.em.0.mac_stats.total_pkts_recvd: 1192738 dev.em.0.mac_stats.good_pkts_recvd: 1191192 dev.em.0.mac_stats.bcast_pkts_recvd: 840327 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 594209 dev.em.0.mac_stats.rx_frames_65_127: 272509 dev.em.0.mac_stats.rx_frames_128_255: 119139 dev.em.0.mac_stats.rx_frames_256_511: 3692 dev.em.0.mac_stats.rx_frames_512_1023: 1217 dev.em.0.mac_stats.rx_frames_1024_1522: 200426 dev.em.0.mac_stats.good_octets_recvd: 391205396 dev.em.0.mac_stats.good_octets_txd: 101322876 dev.em.0.mac_stats.total_pkts_txd: 457538 dev.em.0.mac_stats.good_pkts_txd: 455988 dev.em.0.mac_stats.bcast_pkts_txd: 1812 dev.em.0.mac_stats.mcast_pkts_txd: 3 dev.em.0.mac_stats.tx_frames_64: 1368 dev.em.0.mac_stats.tx_frames_65_127: 187744 dev.em.0.mac_stats.tx_frames_128_255: 218107 dev.em.0.mac_stats.tx_frames_256_511: 10422 dev.em.0.mac_stats.tx_frames_512_1023: 11285 dev.em.0.mac_stats.tx_frames_1024_1522: 27062 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0
-
It looks wrong to me that one NIC on PCI bus 4 uses irq 11 and another NIC on the same bus uses irq 16. (At least such an assignment would be most unlikely on recent Intel hardware where I would expect both to have an irq of 16 or more or both to have 12 or less.)
Please provide the output of the pfSense shell command dmesg so I can see what other devices and irqs are in use of your box. You might also look to see if there is a more recent BIOS that one currently installed, in particular, a BIOS update addressing irq assignments to PCI devices.
-
All fixed :D Jos from applianceshop.eu contacted me and suggested i try disabling ACPI by adding hint.acpi.0.disabled="1" to /boot/loader.conf.
Just in case the dmesg is helpful to anyone else, here it is (with acpi disabled)
Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p4 #0: Thu Jun 2 04:44:19 EDT 2011 sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pnsesrc/src/sys/pfSense_wrap.8.i386 i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Atom(TM) CPU Z530 @ 1.60GHz (1596.00-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x106c2 Family = 6 Model = 1c Stepping = 2 Features=0xbfe9fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mcmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40c3bd <sse3,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,xtpr,pdcm,movbe<br>AMD Features=0x100000 <nx>AMD Features2=0x1 <lahf>TSC: P-state invariant real memory = 1073741824 (1024 MB) avail memory = 1026523136 (978 MB) MPTable: <intel ="" poulsbo="">WARNING: Non-uniform processors. WARNING: Using suboptimal topology. ioapic0: Assuming intbase of 0 ioapic0 <version 2.0="">irqs 0-23 on motherboard wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/ ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in ot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xc0708980, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 inoot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc0708a20, 0) error 1 wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/. wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boooader.conf. module_register_init: MOD_LOAD (wpi_fw, 0xc087b840, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_/. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc0708ac0, 0) error 1 cryptosoft0: <software crypto="">on motherboard padlock0: No ACE support. pcib0: <mptable host-pci="" bridge="">pcibus 0 on motherboard pci0: <pci bus="">on pcib0 vgapci0: <vga-compatible display="">port 0xe880-0xe887 mem 0xdff80000-0xdffffffxd0000000-0xd7ffffff,0xdff60000-0xdff7ffff irq 16 at device 2.0 on pci0 pcib1: <pci-pci bridge="">irq 16 at device 28.0 on pci0 pci1: <pci bus="">on pcib1 pcib2: <mptable pci-pci="" bridge="">at device 0.0 on pci1 pci2: <pci bus="">on pcib2 em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3="">port 0xc880-0xc8bf m0xcfce0000-0xcfcfffff,0xcfcc0000-0xcfcdffff irq 17 at device 1.0 on pci2 em0: [FILTER] rl0: <realtek 10="" 8139="" 100basetx="">port 0xce00-0xceff mem 0xcfc9fc00-0xcfc9fcffq 18 at device 2.0 on pci2 miibus0: <mii bus="">on rl0 rlphy0: <realtek internal="" media="" interface="">PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl0: [ITHREAD] rl1: <realtek 10="" 8139="" 100basetx="">port 0xcd00-0xcdff mem 0xcfc9f800-0xcfc9f8ffq 19 at device 3.0 on pci2 miibus1: <mii bus="">on rl1 rlphy1: <realtek internal="" media="" interface="">PHY 0 on miibus1 rlphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl1: [ITHREAD] em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3="">port 0xc480-0xc4bf m0xcfc60000-0xcfc7ffff,0xcfc40000-0xcfc5ffff irq 16 at device 4.0 on pci2 em1: [FILTER] pcib3: <mptable pci-pci="" bridge="">irq 17 at device 28.1 on pci0 pci3: <pci bus="">on pcib3 re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0xde00-0xdemem 0xcfdff000-0xcfdfffff,0xff9fc000-0xff9fffff irq 17 at device 0.0 on pci3 re0: Using 1 MSI messages re0: Chip rev. 0x28000000 re0: MAC rev. 0x00000000 miibus2: <mii bus="">on re0 rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus2 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseDX, auto re0: [FILTER] uhci0: <uhci (generic)="" usb="" controller="">port 0xe480-0xe49f irq 16 at device 29on pci0 uhci0: [ITHREAD] usbus0: <uhci (generic)="" usb="" controller="">on uhci0 ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xdff5bc00-0xdff5bfff irq 19 aevice 29.7 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0 isab0: <pci-isa bridge="">at device 31.0 on pci0 isa0: <isa bus="">on isab0 atapci0: <intel sch="" udma100="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x,0xffa0-0xffaf at device 31.1 on pci0 ata0: <ata 0="" channel="">on atapci0 ata0: [ITHREAD] cpu0 on motherboard unknown: <pnp0c01>can't assign resources (memory) atrtc0: <at realtime="" clock="">at port 0x70-0x71 irq 8 pnpid PNP0b00 on isa0 uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 pnpid PNP05on isa0 uart0: [FILTER] uart0: console (9600,n,8,1) ppc0: parallel port not found. unknown: <pnp0c01>can't assign resources (memory) Timecounter "TSC" frequency 1595998260 Hz quality 800 Timecounters tick every 10.000 msec IPsec: Initialized Security Association Processing. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: <intel>at usbus0 uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0 ugen1.1: <intel>at usbus1 uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1 ad1: 967MB <cf 20090819="" 1gb="">at ata0-slave PIO4 GEOM: ad1s1: media size does not match label. GEOM: ad1s2: media size does not match label. Root mount waiting for: usbus1 usbus0 uhub0: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 uhub1: 4 ports with 4 removable, self powered Trying to mount root from ufs:/dev/ufs/pfsense0 pflog0: promiscuous mode enabled em0: link state changed to UP rl0: link state changed to UP rl1: link state changed to UP WARNING: pseudo-random number generator used for IPsec processing arp: 10.21.3.22 moved from 58:55:ca:07:0d:59 to 00:23:12:0b:4b:b1 on re0</cf></intel></intel></intel></intel></pnp0c01></at></pnp0c01></ata></intel></isa></pci-isa></ehci></ehci></uhci></uhci></rtl8169s></mii></realtek></pci></mptable></intel(r)></realtek></mii></realtek></realtek></mii></realtek></intel(r)></pci></mptable></pci></pci-pci></vga-compatible></pci></mptable></software></version></intel></lahf></nx></sse3,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,xtpr,pdcm,movbe<br></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mcmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>