Testers needed: LSI MegaRAID controllers

msi

Due to need at work with the helpful guys at BSDPerimeter pfSense 2.1 has recently
gained a much-update 'mfi' driver via 8-STABLE.

This backport added support for quite a lot of the newer Drake Skinny and Thunderbolt cards,
found in pre-built servers. It includes also an updated 'mfiutil' which in most cases drops the need
for linux compat and the proprietary MegaCLI.

If you happen to have a box with a LSI-based RAID controller, I'd be happy to hear from you users if this driver works for you too.

mfi-driver cards are known under the labels of:
* LSI MegaRAID
* Dell PERC
* IBM ServeRAID (older ones like ServeRAID 8k are Adaptec based)

I'll mostly test on a ServeRAID M1115, but have also a M1015 within reach
(both are almost identical to MegaRAID 9240, but with RAID5 locked)

trunglam

Good idea, tomorow i will reply after test

msi

Looking forward to your feedback
If it's a branded card, it would be interesting to know on what LSI board the card is based upon.

So far the M1115 hasn't caused hangs or crashes, I hope this will stays so…

trunglam

Report 2.1 Worked on Raid LSI Dell PERC 5.
Now i'm find for LSI MegaRAID and hope it ok.

shawniverson

Hope this helps.

Testing on LSI MegaRAID SAS 9240-8i on a Lenovo Thinkserver RD430-3064-G2U

Using mfi driver on pfSense-2.1-BETA1-amd64-20121221-0526 snapshot

Also seems well…with a few caveats...

mfi appears to be talking to the individual disks directly and is not talking via the RAID controller (ignores RAID Configuration)

Can install pfsense onto either or both disks with or without gmirror.

Reboot, however, and the UEFI BIOS cannot see the disks as bootable.

I am guessing this is because the UEFI is asking the RAID controller for a bootable device, and since it is basing this off of a RAID Configuration it answers back with no bootable device.

It doesn't appear I can disable the RAID controller BIOS or bypass it in a "SAS" mode. Since the controller is embedded in the mb I am not likely to flash it with a firmware that the manufacturer will not support.

It also doesn't appear that I can change UEFI to a legacy mode either.

I am trying a temporary workaround with a USB stick containing the boot code and necessary boot components. It appears by issuing the mount command manually I can successfully boot to the disks from the USB stick since UEFI recognizes it as a bootable device.

msi

Interesting feedback shawniverson, especially since you have an LSI 9240 based card too.
Personally I did create the RAID in LSI menu in UEFI on the IBM box i have here (x3550 m4), not with mfiutil.
Could you post the output of mfiutil show [adapter | drives | volumes | config] on both pfSense and a stock FreeBSD 9.1 (to see if they differ)?

Though, it would be interesting to use mfiutil to destroy and recreate the array (while in live mode) before you go into the installer on your side…

My IBM box has UEFI too but by default is enabled BIOS compatible mode, here is a document about UEFI for a ThinkServer TS430
(not RD430) about UEFI maybe that helps? http://shop.lenovo.com/ISS_Static/ww/wci/products/us/server/content/pdfs/UEFI%20on%20TS430.pdf

The mfi code that we have in pfSense 2.1 via 8-STABLE is quite close, and contains some fixes even after 9.1 was relased, it also contains the updated mfiutil.

shawniverson

Hello MatSim,

Interestingly, I and vreid473 experimented more with the RAID Configuration.

We found the following to be true:

1) With a RAID Configuration defined and one or more logical volumes intitialized and online, and
2) pfsense installed on one physical drive only without gmirror

The system will successfully boot. This is contrary to my initial post somewhat as it appears that I didn't install it quite right the first time (the RAID Configuration was wiped out after the first attempt with gmirror and I didn't catch it)

However, if a RAID Configuration is defined, and a gmirror is created, the RAID Configuration is effectively wiped out, the disks are no longer online, and the system will not boot.

pfsense appears to be not recognizing the RAID logical volumes and sees only physical volumes via the mfi driver.

I and vreid473 will continue testing and will post more details, including mfiutil information.

vreid473

It may also be worth noting:

On the RD430, it appears that the LSI raid controller does not show up in the UEFI shell – either with or without a virtual disk defined in the raid setup. There are several "unknowns" that show up in the UEFI shell, but nothing listed as LSI or RAID. The RD430, unlike the TS430, also does not have a way to disable UEFI from inside the bios configuration utility.

msi

I wouldn't mix LSI raid and gmirror, I'd say it's either or, but not both.
When installing without gmirror but LSI-defined RAID, are you positive it will install on a single physical drive, not rather the LSI (RAID) logical drive?

If the controller is doing the raid, you should see the logical drives as 'mfid <x>under /dev, the first would be /dev/mfid0.
Can you confirm or deny this? (looking forward to your mfiutil outputs)

Additionally you can see how mfi kicks in and detects PD and LDs during boot - and possibly see problematic things with 'dmesg | more' in live mode.

The vanilla firmware for the LSI 9240 says to contain a UEFI driver in the ROM (check their site) - it would be strange if the Lenovo-branded ROM woludn't (their description is less detailed, so is IBM's)
Maybe you can contact Lenovo and ask why actually this UEFI driver is missing - I guess it shouldn't.

I guess although the firmware branding is different, that the Lenovo card must be close to IBM's ServeRAID M1015 since it has the same hardware RAID5 key, not a software one like the M1115.</x>

vreid473

Hi MatSim,

Below are the output of ls -al /dev, cat /etc/fstab, and dmesg along with some commentary from myself after the command output.

[2.1-BETA1][admin@pfsense.localdomain]/root(6): ls -al /dev
total 5
dr-xr-xr-x   7 root  wheel          512 Dec 23 17:47 .
drwxr-xr-x  19 root  wheel          512 Dec 24 07:36 ..
crw-r-----   1 root  operator    0,  96 Dec 23 17:47 acd0
crw-r--r--   1 root  wheel       0,  35 Dec 23 17:47 acpi
crw-------   1 root  operator    0,  34 Dec 23 17:47 ata
crw-------   1 root  wheel       0,  76 Dec 23 17:47 atkbd0
crw-------   1 root  kmem        0,  24 Dec 23 17:47 audit
crw-------   1 root  wheel       0,   6 Dec 23 17:47 bpf
lrwxr-xr-x   1 root  wheel            3 Dec 23 17:47 bpf0 -> bpf
crw-------   1 root  wheel       0,   5 Dec 23 17:48 console
crw-------   1 root  wheel       0,  75 Dec 23 17:47 consolectl
crw-rw-rw-   1 root  wheel       0,  84 Dec 23 17:47 crypto
crw-rw-rw-   1 root  wheel       0,  33 Dec 23 17:47 ctty
crw-rw----   1 uucp  dialer      0,  81 Dec 23 17:47 cuau0
crw-rw----   1 uucp  dialer      0,  82 Dec 23 17:47 cuau0.init
crw-rw----   1 uucp  dialer      0,  83 Dec 23 17:47 cuau0.lock
crw-rw----   1 uucp  dialer      0,  56 Dec 23 17:47 cuau1
crw-rw----   1 uucp  dialer      0,  57 Dec 23 17:47 cuau1.init
crw-rw----   1 uucp  dialer      0,  58 Dec 23 17:47 cuau1.lock
crw-rw----   1 uucp  dialer      0,  50 Dec 23 17:47 cuau2
crw-rw----   1 uucp  dialer      0,  51 Dec 23 17:47 cuau2.init
crw-rw----   1 uucp  dialer      0,  52 Dec 23 17:47 cuau2.lock
crw-r-----   1 root  operator    0, 107 Dec 23 17:47 da0
crw-r-----   1 root  operator    0, 108 Dec 23 17:47 da0s1
crw-r-----   1 root  operator    0, 109 Dec 23 17:47 da0s1a
crw-r-----   1 root  operator    0, 110 Dec 23 17:47 da0s1b
crw-------   1 root  wheel       0,  19 Dec 23 17:47 dcons
crw-------   1 root  wheel       0,   4 Dec 23 17:47 devctl
cr--------   1 root  wheel       0,  95 Dec 23 17:47 devstat
crw-------   1 root  wheel       0,  20 Dec 23 17:47 dgdb
lrwxr-xr-x   1 root  wheel           11 Dec 23 17:48 dumpdev -> /dev/da0s1b
dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 fd
crw-------   1 root  wheel       0,   8 Dec 23 17:47 fido
crw-r-----   1 root  operator    0,   3 Dec 23 17:47 geom.ctl
crw-------   1 root  wheel       0,  30 Dec 23 17:47 io
lrwxr-xr-x   1 root  wheel            6 Dec 23 17:47 kbd0 -> atkbd0
lrwxr-xr-x   1 root  wheel            7 Dec 23 17:47 kbd1 -> kbdmux0
lrwxr-xr-x   1 root  wheel            5 Dec 23 17:47 kbd2 -> ukbd0
lrwxr-xr-x   1 root  wheel            5 Dec 23 17:47 kbd3 -> ukbd1
crw-------   1 root  wheel       0,   9 Dec 23 17:47 kbdmux0
crw-------   1 root  wheel       0,  28 Dec 23 17:47 klog
crw-r-----   1 root  kmem        0,  12 Dec 23 17:47 kmem
dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 led
crw-r-----   1 root  operator    0, 111 Dec 23 17:48 md0
crw-------   1 root  wheel       0,  88 Dec 23 17:47 mdctl
crw-r-----   1 root  kmem        0,  11 Dec 23 17:47 mem
crw-------   1 root  kmem        0,  21 Dec 23 17:47 nfslock
crw-rw-rw-   1 root  wheel       0,  22 Dec 24 07:36 null
crw-------   1 root  operator    0, 106 Dec 23 17:47 pass0
crw-r--r--   1 root  wheel       0,  25 Dec 23 17:47 pci
crw-rw----   1 root  proxy       0,  85 Dec 23 17:47 pf
crw-rw-rw-   1 root  wheel       0,  32 Dec 23 17:47 ptmx
dr-xr-xr-x   2 root  wheel          512 Dec 24 07:26 pts
crw-rw-rw-   1 root  wheel       0,  26 Dec 23 17:47 random
crw-------   1 root  wheel       0,  31 Dec 23 17:47 speaker
lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stderr -> fd/2
lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stdin -> fd/0
lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stdout -> fd/1
crw-------   1 root  wheel       0,  29 Dec 23 17:47 sysmouse
crw-------   1 root  wheel       0,  78 Dec 23 17:47 ttyu0
crw-------   1 root  wheel       0,  79 Dec 23 17:47 ttyu0.init
crw-------   1 root  wheel       0,  80 Dec 23 17:47 ttyu0.lock
crw-------   1 root  wheel       0,  53 Dec 23 17:47 ttyu1
crw-------   1 root  wheel       0,  54 Dec 23 17:47 ttyu1.init
crw-------   1 root  wheel       0,  55 Dec 23 17:47 ttyu1.lock
crw-------   1 root  wheel       0,  47 Dec 23 17:47 ttyu2
crw-------   1 root  wheel       0,  48 Dec 23 17:47 ttyu2.init
crw-------   1 root  wheel       0,  49 Dec 23 17:47 ttyu2.lock
crw-------   1 root  tty         0,  59 Dec 23 17:48 ttyv0
crw-------   1 root  wheel       0,  60 Dec 23 17:47 ttyv1
crw-------   1 root  wheel       0,  61 Dec 23 17:47 ttyv2
crw-------   1 root  wheel       0,  62 Dec 23 17:47 ttyv3
crw-------   1 root  wheel       0,  63 Dec 23 17:47 ttyv4
crw-------   1 root  wheel       0,  64 Dec 23 17:47 ttyv5
crw-------   1 root  wheel       0,  65 Dec 23 17:47 ttyv6
crw-------   1 root  wheel       0,  66 Dec 23 17:47 ttyv7
crw-------   1 root  wheel       0,  67 Dec 23 17:47 ttyv8
crw-------   1 root  wheel       0,  68 Dec 23 17:47 ttyv9
crw-------   1 root  wheel       0,  69 Dec 23 17:47 ttyva
crw-------   1 root  wheel       0,  70 Dec 23 17:47 ttyvb
crw-------   1 root  wheel       0,  71 Dec 23 17:47 ttyvc
crw-------   1 root  wheel       0,  72 Dec 23 17:47 ttyvd
crw-------   1 root  wheel       0,  73 Dec 23 17:47 ttyve
crw-------   1 root  wheel       0,  74 Dec 23 17:47 ttyvf
dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 ufsid
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen0.1 -> usb/0.1.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen0.2 -> usb/0.2.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.1 -> usb/1.1.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.2 -> usb/1.2.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.3 -> usb/1.3.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.4 -> usb/1.4.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.5 -> usb/1.5.0
lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.6 -> usb/1.6.0
crw-r--r--   1 root  operator    0, 132 Dec 23 17:47 uhid0
crw-------   1 root  wheel       0, 117 Dec 23 17:47 ukbd0
crw-------   1 root  wheel       0, 130 Dec 23 17:47 ukbd1
crw-r--r--   1 root  operator    0, 119 Dec 23 17:47 ums0
crw-r--r--   1 root  operator    0, 120 Dec 23 17:47 ums1
crw-r--r--   1 root  operator    0, 133 Dec 23 17:47 ums2
crw-r--r--   1 root  operator    0, 134 Dec 23 17:47 ums3
lrwxr-xr-x   1 root  wheel            6 Dec 23 17:47 urandom -> random
dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 usb
crw-r--r--   1 root  operator    0,  86 Dec 23 17:47 usbctl
crw-------   1 root  operator    0,  87 Dec 23 17:47 xpt0
crw-rw-rw-   1 root  wheel       0,  23 Dec 23 17:47 zero
[2.1-BETA1][admin@pfsense.localdomain]/root(7):

I notice there are no devices listed as /dev/mfid<x>.

Here is the system's fstab, which also does not reference any mfi devices:

[2.1-BETA1][admin@pfsense.localdomain]/root(8): cat /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/da0s1a             /               ufs     rw              1       1
/dev/da0s1b             none            swap    sw              0       0
[2.1-BETA1][admin@pfsense.localdomain]/root(9):

It looks like pfsense is completely bypassing the LSI raid controller to me. shawniverson and I, however, were able to confirm during testing last night that the system will not boot to a pfsense install if there is not some sort of raid entry for the drive that contains the pfsense on it listed in the LSI mega raid bios setup utility (ctrl + M during boot up to get to it). Currently we are booting this server with 1 physical drive installed on the system with a LSI raid bios entry for a 1 drive raid 0 virtual drive.

There is no mention of mfi in dmesg on the running system. Here's dmesg's output here:

[2.1-BETA1][admin@pfsense.localdomain]/root(4): dmesg
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE-p5 #0: Fri Dec 21 05:55:11 EST 2012
    root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz (2194.72-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 6  Model = 2d  Stepping = 7
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x17bee3ff <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
real memory  = 34359738368 (32768 MB)
avail memory = 33098817536 (31565 MB)
ACPI APIC Table: <lenovo sv-int=""  ="">FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
 cpu4 (AP): APIC ID: 32
 cpu5 (AP): APIC ID: 34
 cpu6 (AP): APIC ID: 36
 cpu7 (AP): APIC ID: 38
ioapic0 <version 2.0="">irqs 0-23 on motherboard
ioapic1 <version 2.0="">irqs 24-47 on motherboard
ioapic2 <version 2.0="">irqs 48-71 on motherboard
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804a3fd0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804a4070, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804a4110, 0) error 1
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <lenovo sv-int="">on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
ACPI Warning: For \\_SB_.SCK0.CP01._CST: Return type mismatch - found Integer, expected Package (20101013/nspredef-1190)
cpu1: invalid _CST package
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
cpu4: <acpi cpu="">on acpi0
cpu5: <acpi cpu="">on acpi0
cpu6: <acpi cpu="">on acpi0
cpu7: <acpi cpu="">on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">irq 26 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
pcib2: <acpi pci-pci="" bridge="">mem 0xdfb00000-0xdfb03fff irq 16 at device 0.0 on pci1
pci2: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">irq 16 at device 8.0 on pci2
pci3: <acpi pci="" bus="">on pcib3
isci0: <intel(r) c600="" series="" chipset="" sas="" controller="">port 0x8100-0x81ff,0x8000-0x80ff mem 0xde8f8000-0xde8fffff,0xde000000-0xde7fffff irq 16 at device 0.0 on pci3
isci0: [ITHREAD]
isci0: [ITHREAD]
isci0: [ITHREAD]
isci0: [ITHREAD]
pci3: <serial bus,="" smbus="">at device 0.3 (no driver attached)
pci3: <serial bus,="" smbus="">at device 0.4 (no driver attached)
pcib4: <acpi pci-pci="" bridge="">irq 26 at device 1.1 on pci0
pci4: <acpi pci="" bus="">on pcib4
pcib5: <acpi pci-pci="" bridge="">irq 40 at device 3.0 on pci0
pci5: <acpi pci="" bus="">on pcib5
pcib6: <acpi pci-pci="" bridge="">irq 40 at device 3.1 on pci0
pci6: <acpi pci="" bus="">on pcib6
pcib7: <acpi pci-pci="" bridge="">irq 40 at device 3.2 on pci0
pci7: <acpi pci="" bus="">on pcib7
pcib8: <acpi pci-pci="" bridge="">irq 40 at device 3.3 on pci0
pci8: <acpi pci="" bus="">on pcib8
pci0: <base peripheral=""> at device 5.0 (no driver attached)
pci0: <base peripheral=""> at device 5.2 (no driver attached)
pci0: <simple comms="">at device 22.0 (no driver attached)
pci0: <simple comms="">at device 22.1 (no driver attached)
ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xdff03000-0xdff033ff irq 16 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
pcib9: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
pci9: <acpi pci="" bus="">on pcib9
igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf900000-0xdf97ffff,0xdf98c000-0xdf98ffff irq 16 at device 0.0 on pci9
igb0: Using MSIX interrupts with 9 vectors
igb0: [ITHREAD]
igb0: Bound queue 0 to cpu 0
igb0: [ITHREAD]
igb0: Bound queue 1 to cpu 1
igb0: [ITHREAD]
igb0: Bound queue 2 to cpu 2
igb0: [ITHREAD]
igb0: Bound queue 3 to cpu 3
igb0: [ITHREAD]
igb0: Bound queue 4 to cpu 4
igb0: [ITHREAD]
igb0: Bound queue 5 to cpu 5
igb0: [ITHREAD]
igb0: Bound queue 6 to cpu 6
igb0: [ITHREAD]
igb0: Bound queue 7 to cpu 7
igb0: [ITHREAD]
igb1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf800000-0xdf87ffff,0xdf988000-0xdf98bfff irq 17 at device 0.1 on pci9
igb1: Using MSIX interrupts with 9 vectors
igb1: [ITHREAD]
igb1: Bound queue 0 to cpu 0
igb1: [ITHREAD]
igb1: Bound queue 1 to cpu 1
igb1: [ITHREAD]
igb1: Bound queue 2 to cpu 2
igb1: [ITHREAD]
igb1: Bound queue 3 to cpu 3
igb1: [ITHREAD]
igb1: Bound queue 4 to cpu 4
igb1: [ITHREAD]
igb1: Bound queue 5 to cpu 5
igb1: [ITHREAD]
igb1: Bound queue 6 to cpu 6
igb1: [ITHREAD]
igb1: Bound queue 7 to cpu 7
igb1: [ITHREAD]
igb2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf780000-0xdf7fffff,0xdf984000-0xdf987fff irq 18 at device 0.2 on pci9
igb2: Using MSIX interrupts with 9 vectors
igb2: [ITHREAD]
igb2: Bound queue 0 to cpu 0
igb2: [ITHREAD]
igb2: Bound queue 1 to cpu 1
igb2: [ITHREAD]
igb2: Bound queue 2 to cpu 2
igb2: [ITHREAD]
igb2: Bound queue 3 to cpu 3
igb2: [ITHREAD]
igb2: Bound queue 4 to cpu 4
igb2: [ITHREAD]
igb2: Bound queue 5 to cpu 5
igb2: [ITHREAD]
igb2: Bound queue 6 to cpu 6
igb2: [ITHREAD]
igb2: Bound queue 7 to cpu 7
igb2: [ITHREAD]
igb3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf700000-0xdf77ffff,0xdf980000-0xdf983fff irq 19 at device 0.3 on pci9
igb3: Using MSIX interrupts with 9 vectors
igb3: [ITHREAD]
igb3: Bound queue 0 to cpu 0
igb3: [ITHREAD]
igb3: Bound queue 1 to cpu 1
igb3: [ITHREAD]
igb3: Bound queue 2 to cpu 2
igb3: [ITHREAD]
igb3: Bound queue 3 to cpu 3
igb3: [ITHREAD]
igb3: Bound queue 4 to cpu 4
igb3: [ITHREAD]
igb3: Bound queue 5 to cpu 5
igb3: [ITHREAD]
igb3: Bound queue 6 to cpu 6
igb3: [ITHREAD]
igb3: Bound queue 7 to cpu 7
igb3: [ITHREAD]
pcib10: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
pci10: <acpi pci="" bus="">on pcib10
igb4: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x7020-0x703f mem 0xdfe20000-0xdfe3ffff,0xdfec4000-0xdfec7fff irq 18 at device 0.0 on pci10
igb4: Using MSIX interrupts with 9 vectors
igb4: [ITHREAD]
igb4: Bound queue 0 to cpu 0
igb4: [ITHREAD]
igb4: Bound queue 1 to cpu 1
igb4: [ITHREAD]
igb4: Bound queue 2 to cpu 2
igb4: [ITHREAD]
igb4: Bound queue 3 to cpu 3
igb4: [ITHREAD]
igb4: Bound queue 4 to cpu 4
igb4: [ITHREAD]
igb4: Bound queue 5 to cpu 5
igb4: [ITHREAD]
igb4: Bound queue 6 to cpu 6
igb4: [ITHREAD]
igb4: Bound queue 7 to cpu 7
igb4: [ITHREAD]
igb5: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x7000-0x701f mem 0xdfe00000-0xdfe1ffff,0xdfec0000-0xdfec3fff irq 19 at device 0.1 on pci10
igb5: Using MSIX interrupts with 9 vectors
igb5: [ITHREAD]
igb5: Bound queue 0 to cpu 0
igb5: [ITHREAD]
igb5: Bound queue 1 to cpu 1
igb5: [ITHREAD]
igb5: Bound queue 2 to cpu 2
igb5: [ITHREAD]
igb5: Bound queue 3 to cpu 3
igb5: [ITHREAD]
igb5: Bound queue 4 to cpu 4
igb5: [ITHREAD]
igb5: Bound queue 5 to cpu 5
igb5: [ITHREAD]
igb5: Bound queue 6 to cpu 6
igb5: [ITHREAD]
igb5: Bound queue 7 to cpu 7
igb5: [ITHREAD]
pcib11: <acpi pci-pci="" bridge="">irq 18 at device 28.6 on pci0
pci12: <acpi pci="" bus="">on pcib11
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x6000-0x601f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 18 at device 0.0 on pci12
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
pcib12: <acpi pci-pci="" bridge="">irq 19 at device 28.7 on pci0
pci13: <acpi pci="" bus="">on pcib12
pcib13: <acpi pci-pci="" bridge="">irq 17 at device 0.0 on pci13
pci14: <acpi pci="" bus="">on pcib13
vgapci0: <vga-compatible display="">port 0x5000-0x507f mem 0xdc000000-0xddffffff,0xdfc00000-0xdfc1ffff irq 19 at device 0.0 on pci14
ehci1: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xdff02000-0xdff023ff irq 23 at device 29.0 on pci0
ehci1: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci1
pcib14: <acpi pci-pci="" bridge="">at device 30.0 on pci0
pci15: <acpi pci="" bus="">on pcib14
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel patsburg="" sata300="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x9090-0x909f,0x9080-0x908f irq 19 at device 31.2 on pci0
ata0: <ata channel="">at channel 0 on atapci0
ata0: [ITHREAD]
ata1: <ata channel="">at channel 1 on atapci0
ata1: [ITHREAD]
pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
atapci1: <intel patsburg="" sata300="" controller="">port 0x9070-0x9077,0x9060-0x9063,0x9050-0x9057,0x9040-0x9043,0x9030-0x903f,0x9020-0x902f irq 19 at device 31.5 on pci0
atapci1: [ITHREAD]
ata2: <ata channel="">at channel 0 on atapci1
ata2: [ITHREAD]
ata3: <ata channel="">at channel 1 on atapci1
ata3: [ITHREAD]
pci0: <dasp>at device 31.6 (no driver attached)
pcib15: <acpi host-pci="" bridge="">on acpi0
pci127: <acpi pci="" bus="">on pcib15
pci127: <base peripheral=""> at device 8.0 (no driver attached)
pci127: <base peripheral=""> at device 8.3 (no driver attached)
pci127: <base peripheral=""> at device 8.4 (no driver attached)
pci127: <base peripheral=""> at device 9.0 (no driver attached)
pci127: <base peripheral=""> at device 9.3 (no driver attached)
pci127: <base peripheral=""> at device 9.4 (no driver attached)
pci127: <base peripheral=""> at device 10.0 (no driver attached)
pci127: <base peripheral=""> at device 10.1 (no driver attached)
pci127: <base peripheral=""> at device 10.2 (no driver attached)
pci127: <base peripheral=""> at device 10.3 (no driver attached)
pci127: <base peripheral=""> at device 11.0 (no driver attached)
pci127: <base peripheral=""> at device 11.3 (no driver attached)
pci127: <base peripheral=""> at device 12.0 (no driver attached)
pci127: <base peripheral=""> at device 12.1 (no driver attached)
pci127: <base peripheral=""> at device 12.6 (no driver attached)
pci127: <base peripheral=""> at device 12.7 (no driver attached)
pci127: <base peripheral=""> at device 13.0 (no driver attached)
pci127: <base peripheral=""> at device 13.1 (no driver attached)
pci127: <base peripheral=""> at device 13.6 (no driver attached)
pci127: <base peripheral=""> at device 14.0 (no driver attached)
pci127: <dasp>at device 14.1 (no driver attached)
pci127: <base peripheral=""> at device 15.0 (no driver attached)
pci127: <base peripheral=""> at device 15.1 (no driver attached)
pci127: <base peripheral=""> at device 15.2 (no driver attached)
pci127: <base peripheral=""> at device 15.3 (no driver attached)
pci127: <base peripheral=""> at device 15.4 (no driver attached)
pci127: <base peripheral=""> at device 15.5 (no driver attached)
pci127: <base peripheral=""> at device 15.6 (no driver attached)
pci127: <base peripheral=""> at device 16.0 (no driver attached)
pci127: <base peripheral=""> at device 16.1 (no driver attached)
pci127: <base peripheral=""> at device 16.2 (no driver attached)
pci127: <base peripheral=""> at device 16.3 (no driver attached)
pci127: <base peripheral=""> at device 16.5 (no driver attached)
pci127: <base peripheral=""> at device 16.6 (no driver attached)
pci127: <base peripheral=""> at device 16.7 (no driver attached)
pci127: <base peripheral=""> at device 17.0 (no driver attached)
pci127: <base peripheral=""> at device 19.0 (no driver attached)
pci127: <dasp>at device 19.1 (no driver attached)
pci127: <dasp>at device 19.4 (no driver attached)
pci127: <dasp>at device 19.5 (no driver attached)
pci127: <base peripheral=""> at device 19.6 (no driver attached)
pcib16: <acpi host-pci="" bridge="">on acpi0
pci128: <acpi pci="" bus="">on pcib16
pcib17: <acpi pci-pci="" bridge="">at device 0.0 on pci128
pci129: <acpi pci="" bus="">on pcib17
pcib18: <acpi pci-pci="" bridge="">irq 50 at device 1.0 on pci128
pci130: <acpi pci="" bus="">on pcib18
pcib19: <acpi pci-pci="" bridge="">irq 50 at device 1.1 on pci128
pci131: <acpi pci="" bus="">on pcib19
pcib20: <acpi pci-pci="" bridge="">irq 64 at device 3.0 on pci128
pci132: <acpi pci="" bus="">on pcib20
igb6: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbe00000-0xfbe7ffff,0xfbe8c000-0xfbe8ffff irq 64 at device 0.0 on pci132
igb6: Using MSIX interrupts with 9 vectors
igb6: [ITHREAD]
igb6: Bound queue 0 to cpu 0
igb6: [ITHREAD]
igb6: Bound queue 1 to cpu 1
igb6: [ITHREAD]
igb6: Bound queue 2 to cpu 2
igb6: [ITHREAD]
igb6: Bound queue 3 to cpu 3
igb6: [ITHREAD]
igb6: Bound queue 4 to cpu 4
igb6: [ITHREAD]
igb6: Bound queue 5 to cpu 5
igb6: [ITHREAD]
igb6: Bound queue 6 to cpu 6
igb6: [ITHREAD]
igb6: Bound queue 7 to cpu 7
igb6: [ITHREAD]
igb7: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbd00000-0xfbd7ffff,0xfbe88000-0xfbe8bfff irq 68 at device 0.1 on pci132
igb7: Using MSIX interrupts with 9 vectors
igb7: [ITHREAD]
igb7: Bound queue 0 to cpu 0
igb7: [ITHREAD]
igb7: Bound queue 1 to cpu 1
igb7: [ITHREAD]
igb7: Bound queue 2 to cpu 2
igb7: [ITHREAD]
igb7: Bound queue 3 to cpu 3
igb7: [ITHREAD]
igb7: Bound queue 4 to cpu 4
igb7: [ITHREAD]
igb7: Bound queue 5 to cpu 5
igb7: [ITHREAD]
igb7: Bound queue 6 to cpu 6
igb7: [ITHREAD]
igb7: Bound queue 7 to cpu 7
igb7: [ITHREAD]
igb8: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbc80000-0xfbcfffff,0xfbe84000-0xfbe87fff irq 69 at device 0.2 on pci132
igb8: Using MSIX interrupts with 9 vectors
igb8: [ITHREAD]
igb8: Bound queue 0 to cpu 0
igb8: [ITHREAD]
igb8: Bound queue 1 to cpu 1
igb8: [ITHREAD]
igb8: Bound queue 2 to cpu 2
igb8: [ITHREAD]
igb8: Bound queue 3 to cpu 3
igb8: [ITHREAD]
igb8: Bound queue 4 to cpu 4
igb8: [ITHREAD]
igb8: Bound queue 5 to cpu 5
igb8: [ITHREAD]
igb8: Bound queue 6 to cpu 6
igb8: [ITHREAD]
igb8: Bound queue 7 to cpu 7
igb8: [ITHREAD]
igb9: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbc00000-0xfbc7ffff,0xfbe80000-0xfbe83fff irq 70 at device 0.3 on pci132
igb9: Using MSIX interrupts with 9 vectors
igb9: [ITHREAD]
igb9: Bound queue 0 to cpu 0
igb9: [ITHREAD]
igb9: Bound queue 1 to cpu 1
igb9: [ITHREAD]
igb9: Bound queue 2 to cpu 2
igb9: [ITHREAD]
igb9: Bound queue 3 to cpu 3
igb9: [ITHREAD]
igb9: Bound queue 4 to cpu 4
igb9: [ITHREAD]
igb9: Bound queue 5 to cpu 5
igb9: [ITHREAD]
igb9: Bound queue 6 to cpu 6
igb9: [ITHREAD]
igb9: Bound queue 7 to cpu 7
igb9: [ITHREAD]
pcib21: <acpi pci-pci="" bridge="">irq 64 at device 3.1 on pci128
pci133: <acpi pci="" bus="">on pcib21
pcib22: <acpi pci-pci="" bridge="">irq 64 at device 3.2 on pci128
pci134: <acpi pci="" bus="">on pcib22
pcib23: <acpi pci-pci="" bridge="">irq 64 at device 3.3 on pci128
pci135: <acpi pci="" bus="">on pcib23
pci128: <base peripheral=""> at device 4.0 (no driver attached)
pci128: <base peripheral=""> at device 4.1 (no driver attached)
pci128: <base peripheral=""> at device 4.2 (no driver attached)
pci128: <base peripheral=""> at device 4.3 (no driver attached)
pci128: <base peripheral=""> at device 4.4 (no driver attached)
pci128: <base peripheral=""> at device 4.5 (no driver attached)
pci128: <base peripheral=""> at device 4.6 (no driver attached)
pci128: <base peripheral=""> at device 4.7 (no driver attached)
pci128: <base peripheral=""> at device 5.0 (no driver attached)
pci128: <base peripheral=""> at device 5.2 (no driver attached)
pcib24: <acpi host-pci="" bridge="">on acpi0
pci255: <acpi pci="" bus="">on pcib24
pci255: <base peripheral=""> at device 8.0 (no driver attached)
pci255: <base peripheral=""> at device 8.3 (no driver attached)
pci255: <base peripheral=""> at device 8.4 (no driver attached)
pci255: <base peripheral=""> at device 9.0 (no driver attached)
pci255: <base peripheral=""> at device 9.3 (no driver attached)
pci255: <base peripheral=""> at device 9.4 (no driver attached)
pci255: <base peripheral=""> at device 10.0 (no driver attached)
pci255: <base peripheral=""> at device 10.1 (no driver attached)
pci255: <base peripheral=""> at device 10.2 (no driver attached)
pci255: <base peripheral=""> at device 10.3 (no driver attached)
pci255: <base peripheral=""> at device 11.0 (no driver attached)
pci255: <base peripheral=""> at device 11.3 (no driver attached)
pci255: <base peripheral=""> at device 12.0 (no driver attached)
pci255: <base peripheral=""> at device 12.1 (no driver attached)
pci255: <base peripheral=""> at device 12.6 (no driver attached)
pci255: <base peripheral=""> at device 12.7 (no driver attached)
pci255: <base peripheral=""> at device 13.0 (no driver attached)
pci255: <base peripheral=""> at device 13.1 (no driver attached)
pci255: <base peripheral=""> at device 13.6 (no driver attached)
pci255: <base peripheral=""> at device 14.0 (no driver attached)
pci255: <dasp>at device 14.1 (no driver attached)
pci255: <base peripheral=""> at device 15.0 (no driver attached)
pci255: <base peripheral=""> at device 15.1 (no driver attached)
pci255: <base peripheral=""> at device 15.2 (no driver attached)
pci255: <base peripheral=""> at device 15.3 (no driver attached)
pci255: <base peripheral=""> at device 15.4 (no driver attached)
pci255: <base peripheral=""> at device 15.5 (no driver attached)
pci255: <base peripheral=""> at device 15.6 (no driver attached)
pci255: <base peripheral=""> at device 16.0 (no driver attached)
pci255: <base peripheral=""> at device 16.1 (no driver attached)
pci255: <base peripheral=""> at device 16.2 (no driver attached)
pci255: <base peripheral=""> at device 16.3 (no driver attached)
pci255: <base peripheral=""> at device 16.5 (no driver attached)
pci255: <base peripheral=""> at device 16.6 (no driver attached)
pci255: <base peripheral=""> at device 16.7 (no driver attached)
pci255: <base peripheral=""> at device 17.0 (no driver attached)
pci255: <base peripheral=""> at device 19.0 (no driver attached)
pci255: <dasp>at device 19.1 (no driver attached)
pci255: <dasp>at device 19.4 (no driver attached)
pci255: <dasp>at device 19.5 (no driver attached)
pci255: <base peripheral=""> at device 19.6 (no driver attached)
acpi_button0: <power button="">on acpi0
acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 3 on acpi0
uart2: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 5 on acpi0
uart1: [FILTER]
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xd8000-0xd8fff,0xd9000-0xd9fff,0xda000-0xdafff,0xdb000-0xdbfff,0xdc000-0xdcfff,0xdd000-0xddfff,0xde000-0xdefff,0xdf000-0xdffff,0xe0000-0xe0fff,0xe1000-0xe1fff,0xe2000-0xe2fff on isa0
atkbd: unable to set the command byte.
sc0: <system console="">at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: unable to set the command byte.
ppc0: cannot reserve I/O port range
uart0: <non-standard ns8250="" class="" uart="" with="" fifos="">at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
uart0: [FILTER]
est0: <enhanced speedstep="" frequency="" control="">on cpu0
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
est1: <enhanced speedstep="" frequency="" control="">on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1e9500001600
device_attach: est1 attach returned 6
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
est2: <enhanced speedstep="" frequency="" control="">on cpu2
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
est3: <enhanced speedstep="" frequency="" control="">on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1e9500001600
device_attach: est3 attach returned 6
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
est4: <enhanced speedstep="" frequency="" control="">on cpu4
p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
est5: <enhanced speedstep="" frequency="" control="">on cpu5
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1e9500001600
device_attach: est5 attach returned 6
p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
est6: <enhanced speedstep="" frequency="" control="">on cpu6
p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
est7: <enhanced speedstep="" frequency="" control="">on cpu7
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 1e9500001600
device_attach: est7 attach returned 6
p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <intel>at usbus0
uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
ugen1.1: <intel>at usbus1
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
acd0: DVDR <plds dvd-rw="" ds8a8sh="" kl31="">at ata0-master UDMA100 SATA 1.5Gb/s
SMP: AP CPU #6 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087="">at usbus0
uhub2: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
ugen1.2: <vendor 0x8087="">at usbus1
uhub3: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen1.3: <no manufacturer="">at usbus1
uhub4: <no 3="" 9="" manufacturer="" gadget="" usb="" hub,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
Root mount waiting for: usbus1
da0 at isci0 bus 0 scbus0 target 0 lun 0
da0: <seagate st3300657ss="" 0008="">Fixed Direct Access SCSI-5 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
uhub4: 5 ports with 5 removable, self powered
ugen1.4: <avocent>at usbus1
ukbd0: <keyboard>on usbus1
kbd2 at ukbd0
ums0: <mouse>on usbus1
ums0: 3 buttons and [Z] coordinates ID=0
ums1: <mouse rel="">on usbus1
ums1: 3 buttons and [XYZ] coordinates ID=0
Root mount waiting for: usbus1
ugen1.5: <vendor 0x0557="">at usbus1
uhub5: <vendor 5="" 9="" 0x0557="" product="" 0x8021,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
Root mount waiting for: usbus1
uhub5: 4 ports with 4 removable, self powered
ugen1.6: <aten international="" co.="" ltd="">at usbus1
ukbd1: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
kbd3 at ukbd1
uhid0: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
ums2: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
ums2: 5 buttons and [XYZ] coordinates ID=0
ums3: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
Trying to mount root from ufs:/dev/da0s1a
igb6: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pid 18459 (ntpd), uid 0: exited on signal 11 (core dumped)
igb6: link state changed to DOWN
pid 12274 (squid), uid 62: exited on signal 6
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
[2.1-BETA1][admin@pfsense.localdomain]/root(5):</aten></aten></aten></aten></aten></vendor></vendor></mouse></mouse></keyboard></avocent></seagate></no></no></vendor></vendor></vendor></vendor></plds></intel></intel></intel></intel></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></non-standard></at></keyboard></generic></system></isa></at></high></power></dasp></dasp></dasp></dasp></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></intel(r)></intel(r)></intel(r)></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></dasp></dasp></dasp></dasp></acpi></acpi></dasp></ata></ata></intel></serial></ata></ata></intel></isa></pci-isa></acpi></acpi></ehci></ehci></vga-compatible></acpi></acpi></acpi></acpi></intel(r)></acpi></acpi></intel(r)></intel(r)></acpi></acpi></intel(r)></intel(r)></intel(r)></intel(r)></acpi></acpi></ehci></ehci></simple></simple></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></serial></serial></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></lenovo></software></version></version></version></lenovo></lahf></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
```</x>

msi

Interesting, I can confirm that you should see the mfi kick in during boot but it doesn't.
(BTW: Could you cut the 'dmesg' and 'ls' output to make yourpost more readable + use code tags http://docs.simplemachines.org/index.php?topic=57.0?) :-)

I guess that you don't see it doesn't attach to a device seen in the output of 'pciconf -lvb' too.
I do have a couple of devices that don't have a driver attached in this box, maybe you can see the a device without driver that looks like the Lenovo RAID card?
Although I'm not into driver hacking, maybe what you're missing is the PCI ID for this board, if you can identify the card in pciconf output, trying to add the ID to the driver code could possibly fix things.

Just guessing: In contrast to IBM which must have taken the same ID as LSI, Lenovo may have taken another PCI ID for their card, the driver doesn't attach because it's unknown.
(cross-flashing to LSI or IBM firmware could be an option - this is not covered by warranty - but largely covered on the net for ZFS builds)

I've once read about an issue that when mfi doesn't kick in, mpt or mps, the drivers for non-RAID HBAs take over (which most often does not work) -
this could explain why the OS may seem to see the single disks instead RAID volumes.

Here is an example for an unknown device and the M1115 which for the mfi(4) does look identical to its LSI counterpart for the OS:


# pciconf -lvb
 <snip!>none0@pci0:0:4:0:	class=0x088000 card=0x3c201014 chip=0x3c208086 rev=0x07 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Sandy Bridge DMA Channel 0'
    class      = base peripheral
    bar   [10] = type Memory, range 64, base 0xfbf1c000, size 16384, enabled
 <snip!>mfi0@pci0:27:0:0:	class=0x010400 card=0x040d1014 chip=0x00731000 rev=0x03 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'MegaRAID SAS 9240'
    class      = mass storage
    subclass   = RAID
    bar   [10] = type I/O Port, range 32, base 0x3000, size 256, enabled
    bar   [14] = type Memory, range 64, base 0x91d40000, size 16384, enabled
    bar   [1c] = type Memory, range 64, base 0x91d00000, size 262144, enabled</snip!></snip!>

Although I did investigate a lot on my own, I was very happy to get the assistance of the BSDPerimeter crew for for the crucial parts.
(without making a shameless plug: I was very thankful for their inputs, if you are doing this for a business, do consider a subscription…) :-)

shawniverson

Matsim,

I have found the root of the problem here after running pciconf. I made a bad assumption. This is not a LSI MegaRAID SAS 9240-8i!

It is an embedded Intel SAS Controller with an LSI MegaRAID software RAID riding on top of it.

Lenovo calls it its "RAID 300" interface.

The Vendor and Device IDs are

0x8086 0x1d68 <–- Intel C606 Patsburg Storage Control Unit

The isci driver (Intel) is in control of the device, not mfi. The isci driver doesn't understand the LSI Software RAID and essentially bypasses it.

Lesson learned: use the pfsense HCL!

msi

Crap, yes this seems to be one of those "shabby" Intel Chipset-integrated things that can run in a Intel or with LSI or even Adaptec firmware mode depending on the OEM.
SuperMicro also makes a couple of boards that have this C606 PCH-integrated - somehow "FakeRAID" stuff IMMV. More expensive boards from them actually have dedicated LSI chip instead of the onboard PCH SAS.

Anyhow, you're not alone also FreeNAS people have been hurt by this: http://forums.freenas.org/archive/index.php/t-8488.html

Since you have the RAID 300 thing, this one definitely is onboard (according to Lenovo RD430 manuals)

Now I'm pretty sure that the RAID 500 would have been the the card you would be looking for and is considered the 9240 card from Lenovo.
This kind of cards are quite affordable, actually the IBM M1015 is quite popular among in DYI ZFS (FreeNAS) builds. Maybe you want to give it a
try, either via the official RAID 500 board or the IBM one. The original LSI cards often are more expensive - but in case of the 9240 come with RAID5 mode unlocked.

The question actually yet is if you want to fight with the PCH-integrated SAS - actually a vanilla FreeBSD 8.3 or better 9.1 could be worth a try to compare - maybe you can
kldload isci over there. OR you go the route and shell out some money for a more "proper" SAS RAID card. Actually I'd even prefer a plain HBA in some situations, but some
server makers like IBM and Lenovo only support and sell you SAS Hardware RAID cards.

Edit: See http://svnweb.freebsd.org/base/release/8.3.0/sys/dev/isci/isci.c?revision=234063&view=markup , the 0x1d68 device should be handled by iscsi, thus trying with a plain FreeBSD 8.3 could be interesting, but YMMV if you want to spend the time with this card or get a true MegaRAID card :-)