Fails to load after Jun 15 update
-
Updated to the 19'th snapshot today.. bad news, pfSense still will not load. Only problem now is that the Jun 12 snapshot that worked for me is no longer available :-[
Here is a full console view of what happens:
http://www.youtube.com/watch?v=5vD7tnvH08s&feature=youtu.beAgain, looks to be doing something with an onboard NIC (bge0) which is Dissabled in the BIOS, Hardware has worked in this state for a long time, and runs properly on any snapshots prior to jun 12-13. Note that it still loads 2.3.1 version of Intel driver… isn't this long outdated?
https://downloadcenter.intel.com/SearchResult.aspx?lang=eng&ProductFamily=Network+Connectivity&ProductLine=Intel%C2%AE+Server+Adapters&ProductProduct=Intel%C2%AE+Gigabit+ET2+Quad+Port+Server+Adapter -
Here is a full console view of what happens:
http://youtu.be/5vD7tnvH08sChromium on Linux tells me the video is currently unavailable. I can't help without more information. Can you provide a few sentences describing the problem in more detail than "will not load"?
-
Clearly it's not disabling correctly in the bios if pfSense can still see it. You are on the latest bios?
Why have you disabled those on-board NICs? Try enabling them and just not using them.Steve
-
Chromium on Linux tells me the video is currently unavailable. I can't help without more information. Can you provide a few sentences describing the problem in more detail than "will not load"?
Dunno why It wouldn't work for you.. wait.. Google owns it ;) Here's an image of the final error screen if you still can't load it.
Here's a URL to the original video that should work http://www.youtube.com/watch?v=5vD7tnvH08s
Clearly it's not disabling correctly in the bios if pfSense can still see it. You are on the latest bios?
Why have you disabled those on-board NICs? Try enabling them and just not using them.Not only is it the latest BIOS its a production tested HP Proliant file server.
Note that any build prior to Jun 12 runs just fine with no BIOS changes, as it has for almost a year now. I am running right now the last beta1 snapshot with no issues, they only start after the Jun 12-13 snapshots. The onboard NICs are disabled for resources since there is a 4 port server nic installed, enabling the dual port nic would bring up total of 6 IRQ just for NICs. The BIOS will not let you separate this many IRQs, some nics will end up with a shared IRQ and an even bigger mess.Don't know what more i can add, as the details are very basic. Machine was up and running doing daily snapshot updates, when i attempt to load any update (or do a clean install) with anything newer than the Jun 12-13 snapshot the linked video is exactly what i see.. machine that worked perfectly before now has issues with detecting the machines resources… Revert to a version Jun 12 or prior, and all works perfectly. Pretty safe to say its not a hardware issue when the hardware hasn't changed, and simply reverting to a previous snapshot fixes it!
This guy here http://forum.pfsense.org/index.php/topic,63333.0.html is having the same issues as well so its likely not a isolated hardware incident. Note that in his case its simply referencing a different resource other than his NIC but the end result is still the same.
My only guess is that somewhere along the line of updating around jun 13-15 there was a change in code that changes the way pfSense accesses the machines local resources. BTW, the Jun 12 build will not detect the disabled NICS on bootup, nor will a windows install. This has also been tested with both AHCI on and with it off too.
Only other guess is that its an issue fixed by Intel long ago with their driver updates since the one pfSense loads is now 6-7 revisions out of date and no longer supported by Intel.
pfSense loads 2.3.1 version when this has been long pulled from Intels supported driver page and replaced with 2.3.7 and 2.3.8. If you watch the video these outdated drivers are indeed being loaded shortly before the machine halts.https://downloadcenter.intel.com/SearchResult.aspx?lang=eng&ProductFamily=Network+Connectivity&ProductLine=Intel%C2%AE+Server+Adapters&ProductProduct=Intel%C2%AE+Gigabit+ET2+Quad+Port+Server+Adapter
-
Hmm, some weirdness indeed.
I note that the crash occours during the system trying to detach the bge driver. If you look at your boot logs from an earlier than 12/6 snapshot does it show the bge driver attempting to attach and then detatching?Steve
-
It does seem to see it but the system just keeps on booting up normally anyways. Its currently 2.1-BETA1 (amd64)
built on Wed Dec 26 06:07:54 EST 2012 since all of the snapshots prior to Jun12-13 are long gone and the newer one was used as a beer coaster.….......
Jun 21 21:04:57 pfSense kernel: pcib6: <acpi pci-pci="" bridge="">at device 28.0 on pci0
Jun 21 21:04:57 pfSense kernel: pci2: <acpi pci="" bus="">on pcib6
Jun 21 21:04:57 pfSense kernel: pcib7: <acpi pci-pci="" bridge="">at device 0.0 on pci2
Jun 21 21:04:57 pfSense kernel: pci3: <acpi pci="" bus="">on pcib7
Jun 21 21:04:57 pfSense kernel: bge0: <hp nc326i="" dual="" port="" pci-e="" gigabit="" adapter,="" asic="" rev.="" 0x009003="">at device 4.0 on pci3
Jun 21 21:04:57 pfSense kernel: bge0: 0x10000 bytes of rid 0x10 res 3 failed (0, 0xffffffffffffffff).
Jun 21 21:04:57 pfSense kernel: bge0: couldn't map memory
Jun 21 21:04:57 pfSense kernel: device_attach: bge0 attach returned 6
Jun 21 21:04:57 pfSense kernel: bge1: <hp nc326i="" dual="" port="" pci-e="" gigabit="" adapter,="" asic="" rev.="" 0x009003="">at device 4.1 on pci3
Jun 21 21:04:57 pfSense kernel: bge1: 0x10000 bytes of rid 0x10 res 3 failed (0, 0xffffffffffffffff).
Jun 21 21:04:57 pfSense kernel: bge1: couldn't map memory
Jun 21 21:04:57 pfSense kernel: device_attach: bge1 attach returned 6</hp></hp>
Jun 21 21:04:57 pfSense kernel: pcib8: <acpi pci-pci="" bridge="">at device 28.2 on pci0
Jun 21 21:04:57 pfSense kernel: pci4: <acpi pci="" bus="">on pcib8
Jun 21 21:04:57 pfSense kernel: pcib9: <acpi pci-pci="" bridge="">at device 28.3 on pci0
Jun 21 21:04:57 pfSense kernel: pci7: <acpi pci="" bus="">on pcib9
Jun 21 21:04:57 pfSense kernel: pcib10: <acpi pci-pci="" bridge="">at device 28.4 on pci0
Jun 21 21:04:57 pfSense kernel: pci10: <acpi pci="" bus="">on pcib10
Jun 21 21:04:57 pfSense kernel: pcib11: <acpi pci-pci="" bridge="">at device 28.5 on pci0
Jun 21 21:04:57 pfSense kernel: pci13: <acpi pci="" bus="">on pcib11
…....</acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi> -
Ah, so this could all be an entirely unrelated regression introduced on the 12th.
Jun 21 21:04:57 pfSense kernel: device_attach: bge1 attach returned 6
Implies the driver failed to attach. Odd that it sees the hardware at all but I guess whatever the bios disables prevents if loading.
I notice that other users in the thread you linked to were seeing 'fatal trap 12' in completely different processes.
One for the devs I think.Steve
-
Aye, sees them in both instances but something changed to start making it have an issue.
Thanks for the help, looks to be a few people with similar issues and as the RC nears completion installing it for some users may be an issue; lets hope it gets resolved now that it is identified here ;D -
My guess from the screenshot of the crash report is that a recent code change in either the bge attach function or the bge detach function has resulted in a section of the detach function assuming a resource allocation request in the attach function always succeeded.
It appears the BIOS "disables" the onboard NICs by allocating their device register a memory address of 0xffffffffffffffff which is probably meant to be "invalid" but I would need to look closely at the code to see how it should recognise that as invalid.
It seems you are wanting to disable the onboard NICs to avoid interrupt sharing - a useful thing in some circumstances. However if some of your NICs are using MSI (Message Signalled Interrupts) you may not be getting interrupt sharing. The startup code might report em2 allocated irq 21 and then em2 using MSI or MSIX and irq 258. That means the device uses irq 258 but doesn't put itself on the irq 21 interupt handler. Some of the "older" Intel NICs say they support MSI but their MSI implementation is broken. I expect the em driver in pfSense 2.1 snapshot builds would have a good knowledge of which devices have working MSI and wouldn't attempt to use MSI on devices on which it is broken.
If you post your startup output and the output of shell command```
vmstat -i -
Startup output:
Jun 21 21:04:57 pfSense syslogd: kernel boot file is /boot/kernel/kernel
Jun 21 21:04:57 pfSense kernel: Copyright 1992-2012 The FreeBSD Project.
Jun 21 21:04:57 pfSense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 21 21:04:57 pfSense kernel: The Regents of the University of California. All rights reserved.
Jun 21 21:04:57 pfSense kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 21 21:04:57 pfSense kernel: FreeBSD 8.3-RELEASE-p5 #0: Wed Dec 26 06:33:36 EST 2012
Jun 21 21:04:57 pfSense kernel: root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
Jun 21 21:04:57 pfSense kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jun 21 21:04:57 pfSense kernel: CPU: Intel(R) Xeon(R) CPU X3210 @ 2.13GHz (2128.01-MHz K8-class CPU)
Jun 21 21:04:57 pfSense kernel: Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11
Jun 21 21:04:57 pfSense kernel: Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Jun 21 21:04:57 pfSense kernel: Features2=0xe3bd <sse3,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm>Jun 21 21:04:57 pfSense kernel: AMD Features=0x20100800 <syscall,nx,lm>Jun 21 21:04:57 pfSense kernel: AMD Features2=0x1 <lahf>Jun 21 21:04:57 pfSense kernel: TSC: P-state invariant
Jun 21 21:04:57 pfSense kernel: real memory = 4294967296 (4096 MB)
Jun 21 21:04:57 pfSense kernel: avail memory = 4102864896 (3912 MB)
Jun 21 21:04:57 pfSense kernel: ACPI APIC Table: <hp ="" proliant="">Jun 21 21:04:57 pfSense kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Jun 21 21:04:57 pfSense kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Jun 21 21:04:57 pfSense kernel: cpu0 (BSP): APIC ID: 0
Jun 21 21:04:57 pfSense kernel: cpu1 (AP): APIC ID: 1
Jun 21 21:04:57 pfSense kernel: cpu2 (AP): APIC ID: 2
Jun 21 21:04:57 pfSense kernel: cpu3 (AP): APIC ID: 3
Jun 21 21:04:57 pfSense kernel: ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20101013/tbfadt-707)
Jun 21 21:04:57 pfSense kernel: ioapic0 <version 2.0="">irqs 0-23 on motherboard
Jun 21 21:04:57 pfSense kernel: wlan: mac acl policy registered
Jun 21 21:04:57 pfSense kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Jun 21 21:04:57 pfSense kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 21 21:04:57 pfSense kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804a3fd0, 0) error 1
Jun 21 21:04:57 pfSense kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Jun 21 21:04:57 pfSense kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 21 21:04:57 pfSense kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804a4070, 0) error 1
Jun 21 21:04:57 pfSense kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Jun 21 21:04:57 pfSense kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Jun 21 21:04:57 pfSense kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804a4110, 0) error 1
Jun 21 21:04:57 pfSense kernel: kbd1 at kbdmux0
Jun 21 21:04:57 pfSense kernel: cryptosoft0: <software crypto="">on motherboard
Jun 21 21:04:57 pfSense kernel: padlock0: No ACE support.
Jun 21 21:04:57 pfSense kernel: acpi0: <hp proliant="">on motherboard
Jun 21 21:04:57 pfSense kernel: acpi0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: acpi0: Power Button (fixed)
Jun 21 21:04:57 pfSense kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
Jun 21 21:04:57 pfSense kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
Jun 21 21:04:57 pfSense kernel: cpu0: <acpi cpu="">on acpi0
Jun 21 21:04:57 pfSense kernel: cpu1: <acpi cpu="">on acpi0
Jun 21 21:04:57 pfSense kernel: cpu2: <acpi cpu="">on acpi0
Jun 21 21:04:57 pfSense kernel: cpu3: <acpi cpu="">on acpi0
Jun 21 21:04:57 pfSense kernel: pcib0: <acpi host-pci="" bridge="">on acpi0
Jun 21 21:04:57 pfSense kernel: pci0: <acpi pci="" bus="">on pcib0
Jun 21 21:04:57 pfSense kernel: pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0
Jun 21 21:04:57 pfSense kernel: pci21: <acpi pci="" bus="">on pcib1
Jun 21 21:04:57 pfSense kernel: pcib2: <acpi pci-pci="" bridge="">at device 0.0 on pci21
Jun 21 21:04:57 pfSense kernel: pci25: <acpi pci="" bus="">on pcib2
Jun 21 21:04:57 pfSense kernel: pcib3: <pci-pci bridge="">at device 2.0 on pci25
Jun 21 21:04:57 pfSense kernel: pci26: <pci bus="">on pcib3
Jun 21 21:04:57 pfSense kernel: igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x4000-0x401f mem 0xfd6e0000-0xfd6fffff,0xfd400000-0xfd5fffff,0xfd3f0000-0xfd3f3fff irq 18 at device 0.0 on pci26
Jun 21 21:04:57 pfSense kernel: igb0: Using MSIX interrupts with 5 vectors
Jun 21 21:04:57 pfSense kernel: igb0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb0: Bound queue 0 to cpu 0
Jun 21 21:04:57 pfSense kernel: igb0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb0: Bound queue 1 to cpu 1
Jun 21 21:04:57 pfSense kernel: igb0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb0: Bound queue 2 to cpu 2
Jun 21 21:04:57 pfSense kernel: igb0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb0: Bound queue 3 to cpu 3
Jun 21 21:04:57 pfSense kernel: igb0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x4020-0x403f mem 0xfd3c0000-0xfd3dffff,0xfd000000-0xfd1fffff,0xfcff0000-0xfcff3fff irq 19 at device 0.1 on pci26
Jun 21 21:04:57 pfSense kernel: igb1: Using MSIX interrupts with 5 vectors
Jun 21 21:04:57 pfSense kernel: igb1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb1: Bound queue 0 to cpu 0
Jun 21 21:04:57 pfSense kernel: igb1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb1: Bound queue 1 to cpu 1
Jun 21 21:04:57 pfSense kernel: igb1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb1: Bound queue 2 to cpu 2
Jun 21 21:04:57 pfSense kernel: igb1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb1: Bound queue 3 to cpu 3
Jun 21 21:04:57 pfSense kernel: igb1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: pcib4: <pci-pci bridge="">at device 4.0 on pci25
Jun 21 21:04:57 pfSense kernel: pci27: <pci bus="">on pcib4
Jun 21 21:04:57 pfSense kernel: igb2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x5000-0x501f mem 0xfdfe0000-0xfdffffff,0xfdc00000-0xfddfffff,0xfdbf0000-0xfdbf3fff irq 16 at device 0.0 on pci27
Jun 21 21:04:57 pfSense kernel: igb2: Using MSIX interrupts with 5 vectors
Jun 21 21:04:57 pfSense kernel: igb2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb2: Bound queue 0 to cpu 0
Jun 21 21:04:57 pfSense kernel: igb2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb2: Bound queue 1 to cpu 1
Jun 21 21:04:57 pfSense kernel: igb2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb2: Bound queue 2 to cpu 2
Jun 21 21:04:57 pfSense kernel: igb2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb2: Bound queue 3 to cpu 3
Jun 21 21:04:57 pfSense kernel: igb2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x5020-0x503f mem 0xfdbc0000-0xfdbdffff,0xfd800000-0xfd9fffff,0xfd7f0000-0xfd7f3fff irq 17 at device 0.1 on pci27
Jun 21 21:04:57 pfSense kernel: igb3: Using MSIX interrupts with 5 vectors
Jun 21 21:04:57 pfSense kernel: igb3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb3: Bound queue 0 to cpu 0
Jun 21 21:04:57 pfSense kernel: igb3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb3: Bound queue 1 to cpu 1
Jun 21 21:04:57 pfSense kernel: igb3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb3: Bound queue 2 to cpu 2
Jun 21 21:04:57 pfSense kernel: igb3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: igb3: Bound queue 3 to cpu 3
Jun 21 21:04:57 pfSense kernel: igb3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: pcib5: <acpi pci-pci="" bridge="">at device 6.0 on pci0
Jun 21 21:04:57 pfSense kernel: pci18: <acpi pci="" bus="">on pcib5
Jun 21 21:04:57 pfSense kernel: pcib6: <acpi pci-pci="" bridge="">at device 28.0 on pci0
Jun 21 21:04:57 pfSense kernel: pci2: <acpi pci="" bus="">on pcib6
Jun 21 21:04:57 pfSense kernel: pcib7: <acpi pci-pci="" bridge="">at device 0.0 on pci2
Jun 21 21:04:57 pfSense kernel: pci3: <acpi pci="" bus="">on pcib7
Jun 21 21:04:57 pfSense kernel: bge0: <hp nc326i="" dual="" port="" pci-e="" gigabit="" adapter,="" asic="" rev.="" 0x009003="">at device 4.0 on pci3
Jun 21 21:04:57 pfSense kernel: bge0: 0x10000 bytes of rid 0x10 res 3 failed (0, 0xffffffffffffffff).
Jun 21 21:04:57 pfSense kernel: bge0: couldn't map memory
Jun 21 21:04:57 pfSense kernel: device_attach: bge0 attach returned 6
Jun 21 21:04:57 pfSense kernel: bge1: <hp nc326i="" dual="" port="" pci-e="" gigabit="" adapter,="" asic="" rev.="" 0x009003="">at device 4.1 on pci3
Jun 21 21:04:57 pfSense kernel: bge1: 0x10000 bytes of rid 0x10 res 3 failed (0, 0xffffffffffffffff).
Jun 21 21:04:57 pfSense kernel: bge1: couldn't map memory
Jun 21 21:04:57 pfSense kernel: device_attach: bge1 attach returned 6
Jun 21 21:04:57 pfSense kernel: pcib8: <acpi pci-pci="" bridge="">at device 28.2 on pci0
Jun 21 21:04:57 pfSense kernel: pci4: <acpi pci="" bus="">on pcib8
Jun 21 21:04:57 pfSense kernel: pcib9: <acpi pci-pci="" bridge="">at device 28.3 on pci0
Jun 21 21:04:57 pfSense kernel: pci7: <acpi pci="" bus="">on pcib9
Jun 21 21:04:57 pfSense kernel: pcib10: <acpi pci-pci="" bridge="">at device 28.4 on pci0
Jun 21 21:04:57 pfSense kernel: pci10: <acpi pci="" bus="">on pcib10
Jun 21 21:04:57 pfSense kernel: pcib11: <acpi pci-pci="" bridge="">at device 28.5 on pci0
Jun 21 21:04:57 pfSense kernel: pci13: <acpi pci="" bus="">on pcib11
Jun 21 21:04:57 pfSense kernel: uhci0: <intel 82801i="" (ich9)="" usb="" controller="">port 0x1000-0x101f irq 21 at device 29.0 on pci0
Jun 21 21:04:57 pfSense kernel: uhci0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus0: <intel 82801i="" (ich9)="" usb="" controller="">on uhci0
Jun 21 21:04:57 pfSense kernel: uhci1: <intel 82801i="" (ich9)="" usb="" controller="">port 0x1020-0x103f irq 21 at device 29.1 on pci0
Jun 21 21:04:57 pfSense kernel: uhci1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus1: <intel 82801i="" (ich9)="" usb="" controller="">on uhci1
Jun 21 21:04:57 pfSense kernel: uhci2: <intel 82801i="" (ich9)="" usb="" controller="">port 0x1040-0x105f irq 21 at device 29.2 on pci0
Jun 21 21:04:57 pfSense kernel: uhci2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus2: <intel 82801i="" (ich9)="" usb="" controller="">on uhci2
Jun 21 21:04:57 pfSense kernel: uhci3: <intel 82801i="" (ich9)="" usb="" controller="">port 0x1060-0x107f irq 21 at device 29.3 on pci0
Jun 21 21:04:57 pfSense kernel: uhci3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus3: <intel 82801i="" (ich9)="" usb="" controller="">on uhci3
Jun 21 21:04:57 pfSense kernel: ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfccf0000-0xfccf03ff irq 21 at device 29.7 on pci0
Jun 21 21:04:57 pfSense kernel: ehci0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus4: EHCI version 1.0
Jun 21 21:04:57 pfSense kernel: usbus4: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">on ehci0
Jun 21 21:04:57 pfSense kernel: pcib12: <acpi pci-pci="" bridge="">at device 30.0 on pci0
Jun 21 21:04:57 pfSense kernel: pci1: <acpi pci="" bus="">on pcib12
Jun 21 21:04:57 pfSense kernel: vgapci0: <vga-compatible display="">port 0x3000-0x30ff mem 0xd8000000-0xdfffffff,0xfcef0000-0xfcefffff irq 21 at device 2.0 on pci1
Jun 21 21:04:57 pfSense kernel: pci1: <base peripheral=""> at device 4.0 (no driver attached)
Jun 21 21:04:57 pfSense kernel: pci1: <base peripheral=""> at device 4.2 (no driver attached)
Jun 21 21:04:57 pfSense kernel: uhci4: <uhci (generic)="" usb="" controller="">port 0x3800-0x381f irq 23 at device 4.4 on pci1
Jun 21 21:04:57 pfSense kernel: uhci4: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: usbus5: <uhci (generic)="" usb="" controller="">on uhci4
Jun 21 21:04:57 pfSense kernel: pci1: <serial bus="">at device 4.6 (no driver attached)
Jun 21 21:04:57 pfSense kernel: isab0: <pci-isa bridge="">at device 31.0 on pci0
Jun 21 21:04:57 pfSense kernel: isa0: <isa bus="">on isab0
Jun 21 21:04:57 pfSense kernel: atapci0: <intel ich9="" sata300="" controller="">port 0x1080-0x1087,0x1088-0x108b,0x1090-0x1097,0x1098-0x109b,0x10a0-0x10af,0x10b0-0x10bf irq 20 at device 31.2 on pci0
Jun 21 21:04:57 pfSense kernel: atapci0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: ata2: <ata channel="">at channel 0 on atapci0
Jun 21 21:04:57 pfSense kernel: ata2: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: ata3: <ata channel="">at channel 1 on atapci0
Jun 21 21:04:57 pfSense kernel: ata3: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: atapci1: <intel ich9="" sata300="" controller="">port 0x10c0-0x10c7,0x10c8-0x10cb,0x10d0-0x10d7,0x10d8-0x10db,0x10e0-0x10ef,0x10f0-0x10ff irq 20 at device 31.5 on pci0
Jun 21 21:04:57 pfSense kernel: atapci1: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: ata4: <ata channel="">at channel 0 on atapci1
Jun 21 21:04:57 pfSense kernel: ata4: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: ata5: <ata channel="">at channel 1 on atapci1
Jun 21 21:04:57 pfSense kernel: ata5: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: acpi_tz0: <thermal zone="">on acpi0
Jun 21 21:04:57 pfSense kernel: acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Jun 21 21:04:57 pfSense kernel: Timecounter "HPET" frequency 14318180 Hz quality 900
Jun 21 21:04:57 pfSense kernel: atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
Jun 21 21:04:57 pfSense kernel: atkbd0: <at keyboard="">irq 1 on atkbdc0
Jun 21 21:04:57 pfSense kernel: kbd0 at atkbd0
Jun 21 21:04:57 pfSense kernel: atkbd0: [GIANT-LOCKED]
Jun 21 21:04:57 pfSense kernel: atkbd0: [ITHREAD]
Jun 21 21:04:57 pfSense kernel: orm0: <isa option="" rom="">at iomem 0xc0000-0xcafff on isa0
Jun 21 21:04:57 pfSense kernel: sc0: <system console="">at flags 0x100 on isa0
Jun 21 21:04:57 pfSense kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jun 21 21:04:57 pfSense kernel: vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jun 21 21:04:57 pfSense kernel: atrtc0: <at real="" time="" clock="">at port 0x70 irq 8 on isa0
Jun 21 21:04:57 pfSense kernel: ppc0: cannot reserve I/O port range
Jun 21 21:04:57 pfSense kernel: est0: <enhanced speedstep="" frequency="" control="">on cpu0
Jun 21 21:04:57 pfSense kernel: est: CPU supports Enhanced Speedstep, but is not recognized.
Jun 21 21:04:57 pfSense kernel: est: cpu_vendor GenuineIntel, msr 825082506000825
Jun 21 21:04:57 pfSense kernel: device_attach: est0 attach returned 6
Jun 21 21:04:57 pfSense kernel: p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
Jun 21 21:04:57 pfSense kernel: est1: <enhanced speedstep="" frequency="" control="">on cpu1
Jun 21 21:04:57 pfSense kernel: est: CPU supports Enhanced Speedstep, but is not recognized.
Jun 21 21:04:57 pfSense kernel: est: cpu_vendor GenuineIntel, msr 825082506000825
Jun 21 21:04:57 pfSense kernel: device_attach: est1 attach returned 6
Jun 21 21:04:57 pfSense kernel: p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
Jun 21 21:04:57 pfSense kernel: est2: <enhanced speedstep="" frequency="" control="">on cpu2
Jun 21 21:04:57 pfSense kernel: est: CPU supports Enhanced Speedstep, but is not recognized.
Jun 21 21:04:57 pfSense kernel: est: cpu_vendor GenuineIntel, msr 825082506000825
Jun 21 21:04:57 pfSense kernel: device_attach: est2 attach returned 6
Jun 21 21:04:57 pfSense kernel: p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
Jun 21 21:04:57 pfSense kernel: est3: <enhanced speedstep="" frequency="" control="">on cpu3
Jun 21 21:04:57 pfSense kernel: est: CPU supports Enhanced Speedstep, but is not recognized.
Jun 21 21:04:57 pfSense kernel: est: cpu_vendor GenuineIntel, msr 825082506000825
Jun 21 21:04:57 pfSense kernel: device_attach: est3 attach returned 6
Jun 21 21:04:57 pfSense kernel: p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
Jun 21 21:04:57 pfSense kernel: Timecounters tick every 1.000 msec
Jun 21 21:04:57 pfSense kernel: IPsec: Initialized Security Association Processing.
Jun 21 21:04:57 pfSense kernel: usbus0: 12Mbps Full Speed USB v1.0
Jun 21 21:04:57 pfSense kernel: usbus1: 12Mbps Full Speed USB v1.0
Jun 21 21:04:57 pfSense kernel: usbus2: 12Mbps Full Speed USB v1.0
Jun 21 21:04:57 pfSense kernel: usbus3: 12Mbps Full Speed USB v1.0
Jun 21 21:04:57 pfSense kernel: usbus4: 480Mbps High Speed USB v2.0
Jun 21 21:04:57 pfSense kernel: usbus5: 12Mbps Full Speed USB v1.0
Jun 21 21:04:57 pfSense kernel: ugen0.1: <intel>at usbus0
Jun 21 21:04:57 pfSense kernel: uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
Jun 21 21:04:57 pfSense kernel: ugen1.1: <intel>at usbus1
Jun 21 21:04:57 pfSense kernel: uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
Jun 21 21:04:57 pfSense kernel: ugen2.1: <intel>at usbus2
Jun 21 21:04:57 pfSense kernel: uhub2: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
Jun 21 21:04:57 pfSense kernel: ugen3.1: <intel>at usbus3
Jun 21 21:04:57 pfSense kernel: uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus3
Jun 21 21:04:57 pfSense kernel: ugen4.1: <intel>at usbus4
Jun 21 21:04:57 pfSense kernel: uhub4: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus4
Jun 21 21:04:57 pfSense kernel: ugen5.1: <0x103c> at usbus5
Jun 21 21:04:57 pfSense kernel: uhub5: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
Jun 21 21:04:57 pfSense kernel: ad4: 57241MB <mushkin mknssdcl60gb-dx="" 361a13f0="">at ata2-master UDMA100 SATA 1.5Gb/s
Jun 21 21:04:57 pfSense kernel: acd0: DVDROM <dv-28e-v c.ab="">at ata5-master UDMA33 SATA 1.5Gb/s
Jun 21 21:04:57 pfSense kernel: SMP: AP CPU #3 Launched!
Jun 21 21:04:57 pfSense kernel: SMP: AP CPU #1 Launched!
Jun 21 21:04:57 pfSense kernel: SMP: AP CPU #2 Launched!
Jun 21 21:04:57 pfSense kernel: Root mount waiting for: usbus5 usbus4 usbus3 usbus2 usbus1 usbus0
Jun 21 21:04:57 pfSense kernel: uhub0: 2 ports with 2 removable, self powered
Jun 21 21:04:57 pfSense kernel: uhub1: 2 ports with 2 removable, self powered
Jun 21 21:04:57 pfSense kernel: uhub2: 2 ports with 2 removable, self powered
Jun 21 21:04:57 pfSense kernel: uhub3: 2 ports with 2 removable, self powered
Jun 21 21:04:57 pfSense kernel: uhub5: 2 ports with 2 removable, self powered
Jun 21 21:04:57 pfSense kernel: ugen5.2: <hp>at usbus5
Jun 21 21:04:57 pfSense kernel: Root mount waiting for: usbus5 usbus4
Jun 21 21:04:57 pfSense kernel: ukbd0: <virtual keyboard="">on usbus5
Jun 21 21:04:57 pfSense kernel: kbd2 at ukbd0
Jun 21 21:04:57 pfSense kernel: ums0: <virtual mouse="">on usbus5
Jun 21 21:04:57 pfSense kernel: ums0: 3 buttons and [XY] coordinates ID=0
Jun 21 21:04:57 pfSense kernel: Root mount waiting for: usbus5 usbus4
Jun 21 21:04:57 pfSense kernel: ugen5.3: <hp>at usbus5
Jun 21 21:04:57 pfSense kernel: uhub6: <virtual hub="">on usbus5
Jun 21 21:04:57 pfSense kernel: Root mount waiting for: usbus5 usbus4
Jun 21 21:04:57 pfSense kernel: uhub6: 7 ports with 7 removable, self powered
Jun 21 21:04:57 pfSense kernel: uhub4: 8 ports with 8 removable, self powered
Jun 21 21:04:57 pfSense kernel: Trying to mount root from ufs:/dev/ad4s1a
Jun 21 21:04:57 pfSense kernel: ugen3.2: <razer>at usbus3
Jun 21 21:04:57 pfSense kernel: ukbd1: <razer 0="" 2="" razer="" lycosa,="" class="" 0,="" rev="" 1.10="" b.62,="" addr="">on usbus3
Jun 21 21:04:57 pfSense kernel: kbd3 at ukbd1
Jun 21 21:04:57 pfSense kernel: uhid0: <razer 0="" 2="" razer="" lycosa,="" class="" 0,="" rev="" 1.10="" b.62,="" addr="">on usbus3
Jun 21 21:04:59 pfSense php: : Restarting/Starting all packages.
Jun 21 21:04:59 pfSense php: : Creating SSL Certificate for this host
Jun 21 21:04:59 pfSense check_reload_status: Syncing firewall
Jun 21 21:04:59 pfSense php: : ROUTING: setting default route to 12.34.56.78
Jun 21 21:04:59 pfSense check_reload_status: Updating all dyndns
Jun 21 21:05:00 pfSense dhclient: XMT: Solicit on igb0, interval 8210ms.
Jun 21 21:05:00 pfSense check_reload_status: Linkup starting igb1
Jun 21 21:05:00 pfSense kernel: igb1: link state changed to UP
Jun 21 21:05:01 pfSense check_reload_status: Restarting ipsec tunnels
Jun 21 21:05:03 pfSense php: : Creating rrd update script
Jun 21 21:05:04 pfSense php: : Restarting/Starting all packages.
Jun 21 21:05:05 pfSense login: login on ttyv0 as root
…......As you can likely tell its a minimal hardware install in attempt for nothing to interfere with the nics. Still interesting it worked fine before June 13… what changed?
Most of these multi IRQ queues are managed via the Outdated Intel driver once its loaded aren't they? How difficult would it be to get it up to date?
This ET2 isn't an old card.. its currently Intel's $4-500 flagship gigabit server NIC... Would current drivers need to be integrated into pfSense or could a user manage to install the drivers provided by Intel without the need of a secondary redhat machine to compile them on? Regardless if it causes this specific issue or not, I still can't stand running drivers outdated by almost 2 years and 7 versions for this card! Rest of the network has its drivers updated DAILY...</razer></razer></razer></virtual></hp></virtual></virtual></hp></dv-28e-v></mushkin></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></at></generic></system></isa></at></keyboard></high></thermal></ata></ata></intel></ata></ata></intel></isa></pci-isa></serial></uhci></uhci></vga-compatible></acpi></acpi></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></hp></hp></acpi></acpi></acpi></acpi></acpi></acpi></intel(r)></intel(r)></pci></pci-pci></intel(r)></intel(r)></pci></pci-pci></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></hp></software></version></hp></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
Your Intel NICs are all allocated MSIs (irq 256 and above) which are exclusive use - they aren't shared with any other device.
Because your bge devices are "disabled" in the BIOS the hardware irq they would use isn't displayed in startup. Sorry I didn't think of that. But the vmstat output shows the interrupts on the "legacy" irqs they might use if they don't use MSI occur at a very low rate (about 1 per second) so overhead of interrupt sharing is a very weak justification for disabling the bge NICs.
Still interesting it worked fine before June 13… what changed?
I'll leave that for a better informed person to answer.
Most of these multi IRQ queues are managed via the Outdated Intel driver once its loaded aren't they? How difficult would it be to get it up to date?
I suspect at this stage of pfSense 2.1 the pfSense developers wouldn't consider "getting the device drivers up to date" to be sufficient reason to change what is already used.
This ET2 isn't an old card.. its currently Intel's $4-500 flagship gigabit server NIC… Would current drivers need to be integrated into pfSense or could a user manage to install the drivers provided by Intel without the need of a secondary redhat machine to compile them on?
Redhat implies Linux. Linux is not FreeBSD. pfSense is based on FreeBSD. But you could build the device drivers on a FreeBSD 8.3 system and copy them over to your pfSense box and configure the box to load them at boot time. You might find that whole process straightforward, you might find it mired in a maze of unexpected complications.
Regardless if it causes this specific issue or not, I still can't stand running drivers outdated by almost 2 years and 7 versions for this card! Rest of the network has its drivers updated DAILY…
That's living on the bleeding edge. Clearly you are not faint hearted. Perhaps you are just the sort of person to embark on what could be a considerable learning adventure.
-
Thanks Bob :D
-
This is getting rather frustrating… 14 days later and still no update or information regarding this situation.
You'd think an issue like this preventing users from running pfSense at all would take some form of priority, but its almost half a month since this has sprung and not even a hint of anybody looking into this despite the documentation presented here as well as the direction of other posters indeed identifying that something changed in the software to cause this.Don't know what more can be done to document this so that it gets resolved so I will put it rather blunt...
Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.
Minor tweaks to pfSense are useless to users if they cant install it… let's get the major stuff out of the way first. This issue was introduced with a snapshot after June 12-13'th, I don't understand the reluctance in going back and actually looking at what changed to cause this and progressing from there. Somebody make the change, therefore somebody should know what they have done....
Forgive the frustration but aside from a few helpful users all of this documentation has fallen on deaf ears so far as the developers go, not a single one has commented in over half a month; which leads me to believe that they have not seen this thread (unlikely) or believe that this is an isolated issue and are not taking it seriously. Ermal,Rbgarga,Jim-p,Phil-davis... anyone seeing this???
-
Looks like someone updated the bge driver between those snapshots.
https://github.com/pfsense/pfsense-tools/commit/4e086e600e71daec708c37c45b265226c5d45186
I'm not sure if that commit actually helped anyone, but if it did create a regression we can back it out.
-
I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.
-
I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.
Thanks Jim, really appreciate it. Will test this immediately when the snapshot becomes available.
-
Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.
Of those who seek out this class of hardware, how many would disable the internal NICs? And why?
-
Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.
Of those who seek out this class of hardware, how many would disable the internal NICs? And why?
The machines are several years old now, can get the best of both worlds with server stability and today's current nics ;) Firmware updates are also much easier to do on your NIC when they are not packaged into an HP or Dell Firmware package! In the Gov. rollout here with the Proliants when we installed the servers we actually swapped out even the hardware nics out for Intels! The onboard ones were dissabled for sure! Some of the later gen DL320's actually come with a version of this 4 port NIC (ET or VT) from Intel directly installed in the machine.
I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.
You fixed it!
Shame they didn't update the Intel drivers instead of the Broadcom ones! ;D -
-
I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.
You fixed it!
Shame they didn't update the Intel drivers instead of the Broadcom ones! ;DWe did. :-)
You have no idea how much better the firewall runs now with these Intel drivers updated. I would say that you can literally SEE a difference. Now the line keeps its maximum 20Mbit upload saturated almost full time, before this update 3/4 saturation was the usual top end; and ya… it boots! I knew the Intel drivers were outdated, but I never considered that it would impact performance THAT much!
Thanks again, and well done!