Testers needed: LSI MegaRAID controllers



  • Due to need at work with the helpful guys at BSDPerimeter pfSense 2.1 has recently
    gained a much-update 'mfi' driver via 8-STABLE.

    This backport added support for quite a lot of the newer Drake Skinny and Thunderbolt cards,
    found in pre-built servers. It includes also an updated 'mfiutil' which in most cases drops the need
    for linux compat and the proprietary MegaCLI.

    If you happen to have a box with a LSI-based RAID controller, I'd be happy to hear from you users if this driver works for you too.

    mfi-driver cards are known under the labels of:
      * LSI MegaRAID
      * Dell PERC
      * IBM ServeRAID (older ones like ServeRAID 8k are Adaptec based)

    I'll mostly test on a ServeRAID M1115, but have also a M1015 within reach
    (both are almost identical to MegaRAID 9240, but with RAID5 locked)



  • Good idea, tomorow i will reply after test



  • Looking forward to your feedback
    If it's a branded card, it would be interesting to know on what LSI board the card is based upon.

    So far the M1115 hasn't caused hangs or crashes, I hope this will stays so…



  • Report 2.1 Worked on Raid LSI Dell PERC 5.
    Now i'm find for LSI MegaRAID and hope it ok.



  • Hope this helps.

    Testing on LSI MegaRAID SAS 9240-8i on a Lenovo Thinkserver RD430-3064-G2U

    Using mfi driver on pfSense-2.1-BETA1-amd64-20121221-0526 snapshot

    Also seems well…with a few caveats...

    mfi appears to be talking to the individual disks directly and is not talking via the RAID controller (ignores RAID Configuration)

    Can install pfsense onto either or both disks with or without gmirror.

    Reboot, however, and the UEFI BIOS cannot see the disks as bootable.

    I am guessing this is because the UEFI is asking the RAID controller for a bootable device, and since it is basing this off of a RAID Configuration it answers back with no bootable device.

    It doesn't appear I can disable the RAID controller BIOS or bypass it in a "SAS" mode.  Since the controller is embedded in the mb I am not likely to flash it with a firmware that the manufacturer will not support.

    It also doesn't appear that I can change UEFI to a legacy mode either.

    I am trying a temporary workaround with a USB stick containing the boot code and necessary boot components.  It appears by issuing the mount command manually I can successfully boot to the disks from the USB stick since UEFI recognizes it as a bootable device.



  • Interesting feedback shawniverson, especially since you have an LSI 9240 based card too.
    Personally I did create the RAID in LSI menu in UEFI on the IBM box i have here (x3550 m4), not with mfiutil.
    Could you post the output of mfiutil show [adapter | drives | volumes | config] on both pfSense and a stock FreeBSD 9.1 (to see if they differ)?

    Though, it would be interesting to use mfiutil to destroy and recreate the array (while in live mode) before you go into the installer on your side…

    My IBM box has UEFI too but by default is enabled BIOS compatible mode, here is a document about UEFI for a ThinkServer TS430
    (not RD430) about UEFI maybe that helps? http://shop.lenovo.com/ISS_Static/ww/wci/products/us/server/content/pdfs/UEFI on TS430.pdf

    The mfi code that we have in pfSense 2.1 via 8-STABLE is quite close, and contains some fixes even after 9.1 was relased, it also contains the updated mfiutil.



  • Hello MatSim,

    Interestingly, I and vreid473 experimented more with the RAID Configuration.

    We found the following to be true:

    1)  With a RAID Configuration defined and one or more logical volumes intitialized and online, and
    2)  pfsense installed on one physical drive only without gmirror

    The system will successfully boot.  This is contrary to my initial post somewhat as it appears that I didn't install it quite right the first time (the RAID Configuration was wiped out after the first attempt with gmirror and I didn't catch it)

    However, if a RAID Configuration is defined, and a gmirror is created, the RAID Configuration is effectively wiped out, the disks are no longer online, and the system will not boot.

    pfsense appears to be not recognizing the RAID logical volumes and sees only physical volumes via the mfi driver.

    I and vreid473 will continue testing and will post more details, including mfiutil information.



  • It may also be worth noting:

    On the RD430, it appears that the LSI raid controller does not show up in the UEFI shell – either with or without a virtual disk defined in the raid setup.  There are several "unknowns" that show up in the UEFI shell, but nothing listed as LSI or RAID.  The RD430, unlike the TS430, also does not have a way to disable UEFI from inside the bios configuration utility.



  • I wouldn't mix LSI raid and gmirror, I'd say it's either or, but not both.
    When installing without gmirror but LSI-defined RAID, are you positive it will install on a single physical drive, not rather the LSI (RAID) logical drive?

    If the controller is doing the raid, you should see the logical drives as 'mfid <x>under /dev, the first would be /dev/mfid0.
    Can you confirm or deny this? (looking forward to your mfiutil outputs)

    Additionally you can see how mfi kicks in and detects PD and LDs during boot  - and possibly see problematic things with 'dmesg | more' in live mode.

    The vanilla firmware for the LSI 9240 says to contain a UEFI driver in the ROM (check their site) - it would be strange if the Lenovo-branded ROM woludn't (their description is less detailed, so is IBM's)
    Maybe you can contact Lenovo and ask why actually this UEFI driver is missing - I guess it shouldn't.

    I guess although the firmware branding is different, that the Lenovo card must be close to IBM's ServeRAID M1015 since it has the same hardware RAID5 key, not a software one like the M1115.</x>



  • Hi MatSim,

    Below are the output of ls -al /dev, cat /etc/fstab, and dmesg along with some commentary from myself after the command output.

    [2.1-BETA1][admin@pfsense.localdomain]/root(6): ls -al /dev
    total 5
    dr-xr-xr-x   7 root  wheel          512 Dec 23 17:47 .
    drwxr-xr-x  19 root  wheel          512 Dec 24 07:36 ..
    crw-r-----   1 root  operator    0,  96 Dec 23 17:47 acd0
    crw-r--r--   1 root  wheel       0,  35 Dec 23 17:47 acpi
    crw-------   1 root  operator    0,  34 Dec 23 17:47 ata
    crw-------   1 root  wheel       0,  76 Dec 23 17:47 atkbd0
    crw-------   1 root  kmem        0,  24 Dec 23 17:47 audit
    crw-------   1 root  wheel       0,   6 Dec 23 17:47 bpf
    lrwxr-xr-x   1 root  wheel            3 Dec 23 17:47 bpf0 -> bpf
    crw-------   1 root  wheel       0,   5 Dec 23 17:48 console
    crw-------   1 root  wheel       0,  75 Dec 23 17:47 consolectl
    crw-rw-rw-   1 root  wheel       0,  84 Dec 23 17:47 crypto
    crw-rw-rw-   1 root  wheel       0,  33 Dec 23 17:47 ctty
    crw-rw----   1 uucp  dialer      0,  81 Dec 23 17:47 cuau0
    crw-rw----   1 uucp  dialer      0,  82 Dec 23 17:47 cuau0.init
    crw-rw----   1 uucp  dialer      0,  83 Dec 23 17:47 cuau0.lock
    crw-rw----   1 uucp  dialer      0,  56 Dec 23 17:47 cuau1
    crw-rw----   1 uucp  dialer      0,  57 Dec 23 17:47 cuau1.init
    crw-rw----   1 uucp  dialer      0,  58 Dec 23 17:47 cuau1.lock
    crw-rw----   1 uucp  dialer      0,  50 Dec 23 17:47 cuau2
    crw-rw----   1 uucp  dialer      0,  51 Dec 23 17:47 cuau2.init
    crw-rw----   1 uucp  dialer      0,  52 Dec 23 17:47 cuau2.lock
    crw-r-----   1 root  operator    0, 107 Dec 23 17:47 da0
    crw-r-----   1 root  operator    0, 108 Dec 23 17:47 da0s1
    crw-r-----   1 root  operator    0, 109 Dec 23 17:47 da0s1a
    crw-r-----   1 root  operator    0, 110 Dec 23 17:47 da0s1b
    crw-------   1 root  wheel       0,  19 Dec 23 17:47 dcons
    crw-------   1 root  wheel       0,   4 Dec 23 17:47 devctl
    cr--------   1 root  wheel       0,  95 Dec 23 17:47 devstat
    crw-------   1 root  wheel       0,  20 Dec 23 17:47 dgdb
    lrwxr-xr-x   1 root  wheel           11 Dec 23 17:48 dumpdev -> /dev/da0s1b
    dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 fd
    crw-------   1 root  wheel       0,   8 Dec 23 17:47 fido
    crw-r-----   1 root  operator    0,   3 Dec 23 17:47 geom.ctl
    crw-------   1 root  wheel       0,  30 Dec 23 17:47 io
    lrwxr-xr-x   1 root  wheel            6 Dec 23 17:47 kbd0 -> atkbd0
    lrwxr-xr-x   1 root  wheel            7 Dec 23 17:47 kbd1 -> kbdmux0
    lrwxr-xr-x   1 root  wheel            5 Dec 23 17:47 kbd2 -> ukbd0
    lrwxr-xr-x   1 root  wheel            5 Dec 23 17:47 kbd3 -> ukbd1
    crw-------   1 root  wheel       0,   9 Dec 23 17:47 kbdmux0
    crw-------   1 root  wheel       0,  28 Dec 23 17:47 klog
    crw-r-----   1 root  kmem        0,  12 Dec 23 17:47 kmem
    dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 led
    crw-r-----   1 root  operator    0, 111 Dec 23 17:48 md0
    crw-------   1 root  wheel       0,  88 Dec 23 17:47 mdctl
    crw-r-----   1 root  kmem        0,  11 Dec 23 17:47 mem
    crw-------   1 root  kmem        0,  21 Dec 23 17:47 nfslock
    crw-rw-rw-   1 root  wheel       0,  22 Dec 24 07:36 null
    crw-------   1 root  operator    0, 106 Dec 23 17:47 pass0
    crw-r--r--   1 root  wheel       0,  25 Dec 23 17:47 pci
    crw-rw----   1 root  proxy       0,  85 Dec 23 17:47 pf
    crw-rw-rw-   1 root  wheel       0,  32 Dec 23 17:47 ptmx
    dr-xr-xr-x   2 root  wheel          512 Dec 24 07:26 pts
    crw-rw-rw-   1 root  wheel       0,  26 Dec 23 17:47 random
    crw-------   1 root  wheel       0,  31 Dec 23 17:47 speaker
    lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stderr -> fd/2
    lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stdin -> fd/0
    lrwxr-xr-x   1 root  wheel            4 Dec 23 17:47 stdout -> fd/1
    crw-------   1 root  wheel       0,  29 Dec 23 17:47 sysmouse
    crw-------   1 root  wheel       0,  78 Dec 23 17:47 ttyu0
    crw-------   1 root  wheel       0,  79 Dec 23 17:47 ttyu0.init
    crw-------   1 root  wheel       0,  80 Dec 23 17:47 ttyu0.lock
    crw-------   1 root  wheel       0,  53 Dec 23 17:47 ttyu1
    crw-------   1 root  wheel       0,  54 Dec 23 17:47 ttyu1.init
    crw-------   1 root  wheel       0,  55 Dec 23 17:47 ttyu1.lock
    crw-------   1 root  wheel       0,  47 Dec 23 17:47 ttyu2
    crw-------   1 root  wheel       0,  48 Dec 23 17:47 ttyu2.init
    crw-------   1 root  wheel       0,  49 Dec 23 17:47 ttyu2.lock
    crw-------   1 root  tty         0,  59 Dec 23 17:48 ttyv0
    crw-------   1 root  wheel       0,  60 Dec 23 17:47 ttyv1
    crw-------   1 root  wheel       0,  61 Dec 23 17:47 ttyv2
    crw-------   1 root  wheel       0,  62 Dec 23 17:47 ttyv3
    crw-------   1 root  wheel       0,  63 Dec 23 17:47 ttyv4
    crw-------   1 root  wheel       0,  64 Dec 23 17:47 ttyv5
    crw-------   1 root  wheel       0,  65 Dec 23 17:47 ttyv6
    crw-------   1 root  wheel       0,  66 Dec 23 17:47 ttyv7
    crw-------   1 root  wheel       0,  67 Dec 23 17:47 ttyv8
    crw-------   1 root  wheel       0,  68 Dec 23 17:47 ttyv9
    crw-------   1 root  wheel       0,  69 Dec 23 17:47 ttyva
    crw-------   1 root  wheel       0,  70 Dec 23 17:47 ttyvb
    crw-------   1 root  wheel       0,  71 Dec 23 17:47 ttyvc
    crw-------   1 root  wheel       0,  72 Dec 23 17:47 ttyvd
    crw-------   1 root  wheel       0,  73 Dec 23 17:47 ttyve
    crw-------   1 root  wheel       0,  74 Dec 23 17:47 ttyvf
    dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 ufsid
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen0.1 -> usb/0.1.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen0.2 -> usb/0.2.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.1 -> usb/1.1.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.2 -> usb/1.2.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.3 -> usb/1.3.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.4 -> usb/1.4.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.5 -> usb/1.5.0
    lrwxr-xr-x   1 root  wheel            9 Dec 23 17:47 ugen1.6 -> usb/1.6.0
    crw-r--r--   1 root  operator    0, 132 Dec 23 17:47 uhid0
    crw-------   1 root  wheel       0, 117 Dec 23 17:47 ukbd0
    crw-------   1 root  wheel       0, 130 Dec 23 17:47 ukbd1
    crw-r--r--   1 root  operator    0, 119 Dec 23 17:47 ums0
    crw-r--r--   1 root  operator    0, 120 Dec 23 17:47 ums1
    crw-r--r--   1 root  operator    0, 133 Dec 23 17:47 ums2
    crw-r--r--   1 root  operator    0, 134 Dec 23 17:47 ums3
    lrwxr-xr-x   1 root  wheel            6 Dec 23 17:47 urandom -> random
    dr-xr-xr-x   2 root  wheel          512 Dec 23 17:47 usb
    crw-r--r--   1 root  operator    0,  86 Dec 23 17:47 usbctl
    crw-------   1 root  operator    0,  87 Dec 23 17:47 xpt0
    crw-rw-rw-   1 root  wheel       0,  23 Dec 23 17:47 zero
    [2.1-BETA1][admin@pfsense.localdomain]/root(7):
    

    I notice there are no devices listed as /dev/mfid<x>.

    Here is the system's fstab, which also does not reference any mfi devices:

    [2.1-BETA1][admin@pfsense.localdomain]/root(8): cat /etc/fstab
    # Device                Mountpoint      FStype  Options         Dump    Pass#
    /dev/da0s1a             /               ufs     rw              1       1
    /dev/da0s1b             none            swap    sw              0       0
    [2.1-BETA1][admin@pfsense.localdomain]/root(9):
    

    It looks like pfsense is completely bypassing the LSI raid controller to me.  shawniverson and I, however, were able to confirm during testing last night that the system will not boot to a pfsense install if there is not some sort of raid entry for the drive that contains the pfsense on it listed in the LSI mega raid bios setup utility (ctrl + M during boot up to get to it).  Currently we are booting this server with 1 physical drive installed on the system with a LSI raid bios entry for a 1 drive raid 0 virtual drive.

    There is no mention of mfi in dmesg on the running system.  Here's dmesg's output here:

    [2.1-BETA1][admin@pfsense.localdomain]/root(4): dmesg
    Copyright (c) 1992-2012 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.3-RELEASE-p5 #0: Fri Dec 21 05:55:11 EST 2012
        root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz (2194.72-MHz K8-class CPU)
      Origin = "GenuineIntel"  Id = 0x206d7  Family = 6  Model = 2d  Stepping = 7
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x17bee3ff <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
    real memory  = 34359738368 (32768 MB)
    avail memory = 33098817536 (31565 MB)
    ACPI APIC Table: <lenovo sv-int=""  ="">FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    FreeBSD/SMP: 2 package(s) x 4 core(s)
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP): APIC ID:  2
     cpu2 (AP): APIC ID:  4
     cpu3 (AP): APIC ID:  6
     cpu4 (AP): APIC ID: 32
     cpu5 (AP): APIC ID: 34
     cpu6 (AP): APIC ID: 36
     cpu7 (AP): APIC ID: 38
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    ioapic1 <version 2.0="">irqs 24-47 on motherboard
    ioapic2 <version 2.0="">irqs 48-71 on motherboard
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804a3fd0, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804a4070, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804a4110, 0) error 1
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <lenovo sv-int="">on motherboard
    acpi0: [ITHREAD]
    acpi0: Power Button (fixed)
    Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    ACPI Warning: For \\_SB_.SCK0.CP01._CST: Return type mismatch - found Integer, expected Package (20101013/nspredef-1190)
    cpu1: invalid _CST package
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    cpu4: <acpi cpu="">on acpi0
    cpu5: <acpi cpu="">on acpi0
    cpu6: <acpi cpu="">on acpi0
    cpu7: <acpi cpu="">on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">irq 26 at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    pcib2: <acpi pci-pci="" bridge="">mem 0xdfb00000-0xdfb03fff irq 16 at device 0.0 on pci1
    pci2: <acpi pci="" bus="">on pcib2
    pcib3: <acpi pci-pci="" bridge="">irq 16 at device 8.0 on pci2
    pci3: <acpi pci="" bus="">on pcib3
    isci0: <intel(r) c600="" series="" chipset="" sas="" controller="">port 0x8100-0x81ff,0x8000-0x80ff mem 0xde8f8000-0xde8fffff,0xde000000-0xde7fffff irq 16 at device 0.0 on pci3
    isci0: [ITHREAD]
    isci0: [ITHREAD]
    isci0: [ITHREAD]
    isci0: [ITHREAD]
    pci3: <serial bus,="" smbus="">at device 0.3 (no driver attached)
    pci3: <serial bus,="" smbus="">at device 0.4 (no driver attached)
    pcib4: <acpi pci-pci="" bridge="">irq 26 at device 1.1 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    pcib5: <acpi pci-pci="" bridge="">irq 40 at device 3.0 on pci0
    pci5: <acpi pci="" bus="">on pcib5
    pcib6: <acpi pci-pci="" bridge="">irq 40 at device 3.1 on pci0
    pci6: <acpi pci="" bus="">on pcib6
    pcib7: <acpi pci-pci="" bridge="">irq 40 at device 3.2 on pci0
    pci7: <acpi pci="" bus="">on pcib7
    pcib8: <acpi pci-pci="" bridge="">irq 40 at device 3.3 on pci0
    pci8: <acpi pci="" bus="">on pcib8
    pci0: <base peripheral=""> at device 5.0 (no driver attached)
    pci0: <base peripheral=""> at device 5.2 (no driver attached)
    pci0: <simple comms="">at device 22.0 (no driver attached)
    pci0: <simple comms="">at device 22.1 (no driver attached)
    ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xdff03000-0xdff033ff irq 16 at device 26.0 on pci0
    ehci0: [ITHREAD]
    usbus0: EHCI version 1.0
    usbus0: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
    pcib9: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci9: <acpi pci="" bus="">on pcib9
    igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf900000-0xdf97ffff,0xdf98c000-0xdf98ffff irq 16 at device 0.0 on pci9
    igb0: Using MSIX interrupts with 9 vectors
    igb0: [ITHREAD]
    igb0: Bound queue 0 to cpu 0
    igb0: [ITHREAD]
    igb0: Bound queue 1 to cpu 1
    igb0: [ITHREAD]
    igb0: Bound queue 2 to cpu 2
    igb0: [ITHREAD]
    igb0: Bound queue 3 to cpu 3
    igb0: [ITHREAD]
    igb0: Bound queue 4 to cpu 4
    igb0: [ITHREAD]
    igb0: Bound queue 5 to cpu 5
    igb0: [ITHREAD]
    igb0: Bound queue 6 to cpu 6
    igb0: [ITHREAD]
    igb0: Bound queue 7 to cpu 7
    igb0: [ITHREAD]
    igb1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf800000-0xdf87ffff,0xdf988000-0xdf98bfff irq 17 at device 0.1 on pci9
    igb1: Using MSIX interrupts with 9 vectors
    igb1: [ITHREAD]
    igb1: Bound queue 0 to cpu 0
    igb1: [ITHREAD]
    igb1: Bound queue 1 to cpu 1
    igb1: [ITHREAD]
    igb1: Bound queue 2 to cpu 2
    igb1: [ITHREAD]
    igb1: Bound queue 3 to cpu 3
    igb1: [ITHREAD]
    igb1: Bound queue 4 to cpu 4
    igb1: [ITHREAD]
    igb1: Bound queue 5 to cpu 5
    igb1: [ITHREAD]
    igb1: Bound queue 6 to cpu 6
    igb1: [ITHREAD]
    igb1: Bound queue 7 to cpu 7
    igb1: [ITHREAD]
    igb2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf780000-0xdf7fffff,0xdf984000-0xdf987fff irq 18 at device 0.2 on pci9
    igb2: Using MSIX interrupts with 9 vectors
    igb2: [ITHREAD]
    igb2: Bound queue 0 to cpu 0
    igb2: [ITHREAD]
    igb2: Bound queue 1 to cpu 1
    igb2: [ITHREAD]
    igb2: Bound queue 2 to cpu 2
    igb2: [ITHREAD]
    igb2: Bound queue 3 to cpu 3
    igb2: [ITHREAD]
    igb2: Bound queue 4 to cpu 4
    igb2: [ITHREAD]
    igb2: Bound queue 5 to cpu 5
    igb2: [ITHREAD]
    igb2: Bound queue 6 to cpu 6
    igb2: [ITHREAD]
    igb2: Bound queue 7 to cpu 7
    igb2: [ITHREAD]
    igb3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xdf700000-0xdf77ffff,0xdf980000-0xdf983fff irq 19 at device 0.3 on pci9
    igb3: Using MSIX interrupts with 9 vectors
    igb3: [ITHREAD]
    igb3: Bound queue 0 to cpu 0
    igb3: [ITHREAD]
    igb3: Bound queue 1 to cpu 1
    igb3: [ITHREAD]
    igb3: Bound queue 2 to cpu 2
    igb3: [ITHREAD]
    igb3: Bound queue 3 to cpu 3
    igb3: [ITHREAD]
    igb3: Bound queue 4 to cpu 4
    igb3: [ITHREAD]
    igb3: Bound queue 5 to cpu 5
    igb3: [ITHREAD]
    igb3: Bound queue 6 to cpu 6
    igb3: [ITHREAD]
    igb3: Bound queue 7 to cpu 7
    igb3: [ITHREAD]
    pcib10: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci10: <acpi pci="" bus="">on pcib10
    igb4: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x7020-0x703f mem 0xdfe20000-0xdfe3ffff,0xdfec4000-0xdfec7fff irq 18 at device 0.0 on pci10
    igb4: Using MSIX interrupts with 9 vectors
    igb4: [ITHREAD]
    igb4: Bound queue 0 to cpu 0
    igb4: [ITHREAD]
    igb4: Bound queue 1 to cpu 1
    igb4: [ITHREAD]
    igb4: Bound queue 2 to cpu 2
    igb4: [ITHREAD]
    igb4: Bound queue 3 to cpu 3
    igb4: [ITHREAD]
    igb4: Bound queue 4 to cpu 4
    igb4: [ITHREAD]
    igb4: Bound queue 5 to cpu 5
    igb4: [ITHREAD]
    igb4: Bound queue 6 to cpu 6
    igb4: [ITHREAD]
    igb4: Bound queue 7 to cpu 7
    igb4: [ITHREAD]
    igb5: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">port 0x7000-0x701f mem 0xdfe00000-0xdfe1ffff,0xdfec0000-0xdfec3fff irq 19 at device 0.1 on pci10
    igb5: Using MSIX interrupts with 9 vectors
    igb5: [ITHREAD]
    igb5: Bound queue 0 to cpu 0
    igb5: [ITHREAD]
    igb5: Bound queue 1 to cpu 1
    igb5: [ITHREAD]
    igb5: Bound queue 2 to cpu 2
    igb5: [ITHREAD]
    igb5: Bound queue 3 to cpu 3
    igb5: [ITHREAD]
    igb5: Bound queue 4 to cpu 4
    igb5: [ITHREAD]
    igb5: Bound queue 5 to cpu 5
    igb5: [ITHREAD]
    igb5: Bound queue 6 to cpu 6
    igb5: [ITHREAD]
    igb5: Bound queue 7 to cpu 7
    igb5: [ITHREAD]
    pcib11: <acpi pci-pci="" bridge="">irq 18 at device 28.6 on pci0
    pci12: <acpi pci="" bus="">on pcib11
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x6000-0x601f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 18 at device 0.0 on pci12
    em0: Using MSIX interrupts with 3 vectors
    em0: [ITHREAD]
    em0: [ITHREAD]
    em0: [ITHREAD]
    pcib12: <acpi pci-pci="" bridge="">irq 19 at device 28.7 on pci0
    pci13: <acpi pci="" bus="">on pcib12
    pcib13: <acpi pci-pci="" bridge="">irq 17 at device 0.0 on pci13
    pci14: <acpi pci="" bus="">on pcib13
    vgapci0: <vga-compatible display="">port 0x5000-0x507f mem 0xdc000000-0xddffffff,0xdfc00000-0xdfc1ffff irq 19 at device 0.0 on pci14
    ehci1: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xdff02000-0xdff023ff irq 23 at device 29.0 on pci0
    ehci1: [ITHREAD]
    usbus1: EHCI version 1.0
    usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci1
    pcib14: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci15: <acpi pci="" bus="">on pcib14
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel patsburg="" sata300="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x9090-0x909f,0x9080-0x908f irq 19 at device 31.2 on pci0
    ata0: <ata channel="">at channel 0 on atapci0
    ata0: [ITHREAD]
    ata1: <ata channel="">at channel 1 on atapci0
    ata1: [ITHREAD]
    pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
    atapci1: <intel patsburg="" sata300="" controller="">port 0x9070-0x9077,0x9060-0x9063,0x9050-0x9057,0x9040-0x9043,0x9030-0x903f,0x9020-0x902f irq 19 at device 31.5 on pci0
    atapci1: [ITHREAD]
    ata2: <ata channel="">at channel 0 on atapci1
    ata2: [ITHREAD]
    ata3: <ata channel="">at channel 1 on atapci1
    ata3: [ITHREAD]
    pci0: <dasp>at device 31.6 (no driver attached)
    pcib15: <acpi host-pci="" bridge="">on acpi0
    pci127: <acpi pci="" bus="">on pcib15
    pci127: <base peripheral=""> at device 8.0 (no driver attached)
    pci127: <base peripheral=""> at device 8.3 (no driver attached)
    pci127: <base peripheral=""> at device 8.4 (no driver attached)
    pci127: <base peripheral=""> at device 9.0 (no driver attached)
    pci127: <base peripheral=""> at device 9.3 (no driver attached)
    pci127: <base peripheral=""> at device 9.4 (no driver attached)
    pci127: <base peripheral=""> at device 10.0 (no driver attached)
    pci127: <base peripheral=""> at device 10.1 (no driver attached)
    pci127: <base peripheral=""> at device 10.2 (no driver attached)
    pci127: <base peripheral=""> at device 10.3 (no driver attached)
    pci127: <base peripheral=""> at device 11.0 (no driver attached)
    pci127: <base peripheral=""> at device 11.3 (no driver attached)
    pci127: <base peripheral=""> at device 12.0 (no driver attached)
    pci127: <base peripheral=""> at device 12.1 (no driver attached)
    pci127: <base peripheral=""> at device 12.6 (no driver attached)
    pci127: <base peripheral=""> at device 12.7 (no driver attached)
    pci127: <base peripheral=""> at device 13.0 (no driver attached)
    pci127: <base peripheral=""> at device 13.1 (no driver attached)
    pci127: <base peripheral=""> at device 13.6 (no driver attached)
    pci127: <base peripheral=""> at device 14.0 (no driver attached)
    pci127: <dasp>at device 14.1 (no driver attached)
    pci127: <base peripheral=""> at device 15.0 (no driver attached)
    pci127: <base peripheral=""> at device 15.1 (no driver attached)
    pci127: <base peripheral=""> at device 15.2 (no driver attached)
    pci127: <base peripheral=""> at device 15.3 (no driver attached)
    pci127: <base peripheral=""> at device 15.4 (no driver attached)
    pci127: <base peripheral=""> at device 15.5 (no driver attached)
    pci127: <base peripheral=""> at device 15.6 (no driver attached)
    pci127: <base peripheral=""> at device 16.0 (no driver attached)
    pci127: <base peripheral=""> at device 16.1 (no driver attached)
    pci127: <base peripheral=""> at device 16.2 (no driver attached)
    pci127: <base peripheral=""> at device 16.3 (no driver attached)
    pci127: <base peripheral=""> at device 16.5 (no driver attached)
    pci127: <base peripheral=""> at device 16.6 (no driver attached)
    pci127: <base peripheral=""> at device 16.7 (no driver attached)
    pci127: <base peripheral=""> at device 17.0 (no driver attached)
    pci127: <base peripheral=""> at device 19.0 (no driver attached)
    pci127: <dasp>at device 19.1 (no driver attached)
    pci127: <dasp>at device 19.4 (no driver attached)
    pci127: <dasp>at device 19.5 (no driver attached)
    pci127: <base peripheral=""> at device 19.6 (no driver attached)
    pcib16: <acpi host-pci="" bridge="">on acpi0
    pci128: <acpi pci="" bus="">on pcib16
    pcib17: <acpi pci-pci="" bridge="">at device 0.0 on pci128
    pci129: <acpi pci="" bus="">on pcib17
    pcib18: <acpi pci-pci="" bridge="">irq 50 at device 1.0 on pci128
    pci130: <acpi pci="" bus="">on pcib18
    pcib19: <acpi pci-pci="" bridge="">irq 50 at device 1.1 on pci128
    pci131: <acpi pci="" bus="">on pcib19
    pcib20: <acpi pci-pci="" bridge="">irq 64 at device 3.0 on pci128
    pci132: <acpi pci="" bus="">on pcib20
    igb6: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbe00000-0xfbe7ffff,0xfbe8c000-0xfbe8ffff irq 64 at device 0.0 on pci132
    igb6: Using MSIX interrupts with 9 vectors
    igb6: [ITHREAD]
    igb6: Bound queue 0 to cpu 0
    igb6: [ITHREAD]
    igb6: Bound queue 1 to cpu 1
    igb6: [ITHREAD]
    igb6: Bound queue 2 to cpu 2
    igb6: [ITHREAD]
    igb6: Bound queue 3 to cpu 3
    igb6: [ITHREAD]
    igb6: Bound queue 4 to cpu 4
    igb6: [ITHREAD]
    igb6: Bound queue 5 to cpu 5
    igb6: [ITHREAD]
    igb6: Bound queue 6 to cpu 6
    igb6: [ITHREAD]
    igb6: Bound queue 7 to cpu 7
    igb6: [ITHREAD]
    igb7: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbd00000-0xfbd7ffff,0xfbe88000-0xfbe8bfff irq 68 at device 0.1 on pci132
    igb7: Using MSIX interrupts with 9 vectors
    igb7: [ITHREAD]
    igb7: Bound queue 0 to cpu 0
    igb7: [ITHREAD]
    igb7: Bound queue 1 to cpu 1
    igb7: [ITHREAD]
    igb7: Bound queue 2 to cpu 2
    igb7: [ITHREAD]
    igb7: Bound queue 3 to cpu 3
    igb7: [ITHREAD]
    igb7: Bound queue 4 to cpu 4
    igb7: [ITHREAD]
    igb7: Bound queue 5 to cpu 5
    igb7: [ITHREAD]
    igb7: Bound queue 6 to cpu 6
    igb7: [ITHREAD]
    igb7: Bound queue 7 to cpu 7
    igb7: [ITHREAD]
    igb8: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbc80000-0xfbcfffff,0xfbe84000-0xfbe87fff irq 69 at device 0.2 on pci132
    igb8: Using MSIX interrupts with 9 vectors
    igb8: [ITHREAD]
    igb8: Bound queue 0 to cpu 0
    igb8: [ITHREAD]
    igb8: Bound queue 1 to cpu 1
    igb8: [ITHREAD]
    igb8: Bound queue 2 to cpu 2
    igb8: [ITHREAD]
    igb8: Bound queue 3 to cpu 3
    igb8: [ITHREAD]
    igb8: Bound queue 4 to cpu 4
    igb8: [ITHREAD]
    igb8: Bound queue 5 to cpu 5
    igb8: [ITHREAD]
    igb8: Bound queue 6 to cpu 6
    igb8: [ITHREAD]
    igb8: Bound queue 7 to cpu 7
    igb8: [ITHREAD]
    igb9: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">mem 0xfbc00000-0xfbc7ffff,0xfbe80000-0xfbe83fff irq 70 at device 0.3 on pci132
    igb9: Using MSIX interrupts with 9 vectors
    igb9: [ITHREAD]
    igb9: Bound queue 0 to cpu 0
    igb9: [ITHREAD]
    igb9: Bound queue 1 to cpu 1
    igb9: [ITHREAD]
    igb9: Bound queue 2 to cpu 2
    igb9: [ITHREAD]
    igb9: Bound queue 3 to cpu 3
    igb9: [ITHREAD]
    igb9: Bound queue 4 to cpu 4
    igb9: [ITHREAD]
    igb9: Bound queue 5 to cpu 5
    igb9: [ITHREAD]
    igb9: Bound queue 6 to cpu 6
    igb9: [ITHREAD]
    igb9: Bound queue 7 to cpu 7
    igb9: [ITHREAD]
    pcib21: <acpi pci-pci="" bridge="">irq 64 at device 3.1 on pci128
    pci133: <acpi pci="" bus="">on pcib21
    pcib22: <acpi pci-pci="" bridge="">irq 64 at device 3.2 on pci128
    pci134: <acpi pci="" bus="">on pcib22
    pcib23: <acpi pci-pci="" bridge="">irq 64 at device 3.3 on pci128
    pci135: <acpi pci="" bus="">on pcib23
    pci128: <base peripheral=""> at device 4.0 (no driver attached)
    pci128: <base peripheral=""> at device 4.1 (no driver attached)
    pci128: <base peripheral=""> at device 4.2 (no driver attached)
    pci128: <base peripheral=""> at device 4.3 (no driver attached)
    pci128: <base peripheral=""> at device 4.4 (no driver attached)
    pci128: <base peripheral=""> at device 4.5 (no driver attached)
    pci128: <base peripheral=""> at device 4.6 (no driver attached)
    pci128: <base peripheral=""> at device 4.7 (no driver attached)
    pci128: <base peripheral=""> at device 5.0 (no driver attached)
    pci128: <base peripheral=""> at device 5.2 (no driver attached)
    pcib24: <acpi host-pci="" bridge="">on acpi0
    pci255: <acpi pci="" bus="">on pcib24
    pci255: <base peripheral=""> at device 8.0 (no driver attached)
    pci255: <base peripheral=""> at device 8.3 (no driver attached)
    pci255: <base peripheral=""> at device 8.4 (no driver attached)
    pci255: <base peripheral=""> at device 9.0 (no driver attached)
    pci255: <base peripheral=""> at device 9.3 (no driver attached)
    pci255: <base peripheral=""> at device 9.4 (no driver attached)
    pci255: <base peripheral=""> at device 10.0 (no driver attached)
    pci255: <base peripheral=""> at device 10.1 (no driver attached)
    pci255: <base peripheral=""> at device 10.2 (no driver attached)
    pci255: <base peripheral=""> at device 10.3 (no driver attached)
    pci255: <base peripheral=""> at device 11.0 (no driver attached)
    pci255: <base peripheral=""> at device 11.3 (no driver attached)
    pci255: <base peripheral=""> at device 12.0 (no driver attached)
    pci255: <base peripheral=""> at device 12.1 (no driver attached)
    pci255: <base peripheral=""> at device 12.6 (no driver attached)
    pci255: <base peripheral=""> at device 12.7 (no driver attached)
    pci255: <base peripheral=""> at device 13.0 (no driver attached)
    pci255: <base peripheral=""> at device 13.1 (no driver attached)
    pci255: <base peripheral=""> at device 13.6 (no driver attached)
    pci255: <base peripheral=""> at device 14.0 (no driver attached)
    pci255: <dasp>at device 14.1 (no driver attached)
    pci255: <base peripheral=""> at device 15.0 (no driver attached)
    pci255: <base peripheral=""> at device 15.1 (no driver attached)
    pci255: <base peripheral=""> at device 15.2 (no driver attached)
    pci255: <base peripheral=""> at device 15.3 (no driver attached)
    pci255: <base peripheral=""> at device 15.4 (no driver attached)
    pci255: <base peripheral=""> at device 15.5 (no driver attached)
    pci255: <base peripheral=""> at device 15.6 (no driver attached)
    pci255: <base peripheral=""> at device 16.0 (no driver attached)
    pci255: <base peripheral=""> at device 16.1 (no driver attached)
    pci255: <base peripheral=""> at device 16.2 (no driver attached)
    pci255: <base peripheral=""> at device 16.3 (no driver attached)
    pci255: <base peripheral=""> at device 16.5 (no driver attached)
    pci255: <base peripheral=""> at device 16.6 (no driver attached)
    pci255: <base peripheral=""> at device 16.7 (no driver attached)
    pci255: <base peripheral=""> at device 17.0 (no driver attached)
    pci255: <base peripheral=""> at device 19.0 (no driver attached)
    pci255: <dasp>at device 19.1 (no driver attached)
    pci255: <dasp>at device 19.4 (no driver attached)
    pci255: <dasp>at device 19.5 (no driver attached)
    pci255: <base peripheral=""> at device 19.6 (no driver attached)
    acpi_button0: <power button="">on acpi0
    acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 900
    uart2: <16550 or compatible> port 0x3e8-0x3ef irq 3 on acpi0
    uart2: [FILTER]
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 5 on acpi0
    uart1: [FILTER]
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xd8000-0xd8fff,0xd9000-0xd9fff,0xda000-0xdafff,0xdb000-0xdbfff,0xdc000-0xdcfff,0xdd000-0xddfff,0xde000-0xdefff,0xdf000-0xdffff,0xe0000-0xe0fff,0xe1000-0xe1fff,0xe2000-0xe2fff on isa0
    atkbd: unable to set the command byte.
    sc0: <system console="">at flags 0x100 on isa0
    sc0: CGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
    atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    atkbd0: [ITHREAD]
    psm0: unable to set the command byte.
    ppc0: cannot reserve I/O port range
    uart0: <non-standard ns8250="" class="" uart="" with="" fifos="">at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
    uart0: [FILTER]
    est0: <enhanced speedstep="" frequency="" control="">on cpu0
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    est1: <enhanced speedstep="" frequency="" control="">on cpu1
    est: CPU supports Enhanced Speedstep, but is not recognized.
    est: cpu_vendor GenuineIntel, msr 1e9500001600
    device_attach: est1 attach returned 6
    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
    est2: <enhanced speedstep="" frequency="" control="">on cpu2
    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
    est3: <enhanced speedstep="" frequency="" control="">on cpu3
    est: CPU supports Enhanced Speedstep, but is not recognized.
    est: cpu_vendor GenuineIntel, msr 1e9500001600
    device_attach: est3 attach returned 6
    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
    est4: <enhanced speedstep="" frequency="" control="">on cpu4
    p4tcc4: <cpu frequency="" thermal="" control="">on cpu4
    est5: <enhanced speedstep="" frequency="" control="">on cpu5
    est: CPU supports Enhanced Speedstep, but is not recognized.
    est: cpu_vendor GenuineIntel, msr 1e9500001600
    device_attach: est5 attach returned 6
    p4tcc5: <cpu frequency="" thermal="" control="">on cpu5
    est6: <enhanced speedstep="" frequency="" control="">on cpu6
    p4tcc6: <cpu frequency="" thermal="" control="">on cpu6
    est7: <enhanced speedstep="" frequency="" control="">on cpu7
    est: CPU supports Enhanced Speedstep, but is not recognized.
    est: cpu_vendor GenuineIntel, msr 1e9500001600
    device_attach: est7 attach returned 6
    p4tcc7: <cpu frequency="" thermal="" control="">on cpu7
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    usbus0: 480Mbps High Speed USB v2.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: <intel>at usbus0
    uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <intel>at usbus1
    uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
    acd0: DVDR <plds dvd-rw="" ds8a8sh="" kl31="">at ata0-master UDMA100 SATA 1.5Gb/s
    SMP: AP CPU #6 Launched!
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #7 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #4 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #5 Launched!
    Root mount waiting for: usbus1 usbus0
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    Root mount waiting for: usbus1 usbus0
    ugen0.2: <vendor 0x8087="">at usbus0
    uhub2: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
    ugen1.2: <vendor 0x8087="">at usbus1
    uhub3: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
    Root mount waiting for: usbus1 usbus0
    uhub2: 6 ports with 6 removable, self powered
    uhub3: 8 ports with 8 removable, self powered
    ugen1.3: <no manufacturer="">at usbus1
    uhub4: <no 3="" 9="" manufacturer="" gadget="" usb="" hub,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
    Root mount waiting for: usbus1
    da0 at isci0 bus 0 scbus0 target 0 lun 0
    da0: <seagate st3300657ss="" 0008="">Fixed Direct Access SCSI-5 device
    da0: 300.000MB/s transfers
    da0: Command Queueing enabled
    da0: 286102MB (585937500 512 byte sectors: 255H 63S/T 36472C)
    uhub4: 5 ports with 5 removable, self powered
    ugen1.4: <avocent>at usbus1
    ukbd0: <keyboard>on usbus1
    kbd2 at ukbd0
    ums0: <mouse>on usbus1
    ums0: 3 buttons and [Z] coordinates ID=0
    ums1: <mouse rel="">on usbus1
    ums1: 3 buttons and [XYZ] coordinates ID=0
    Root mount waiting for: usbus1
    ugen1.5: <vendor 0x0557="">at usbus1
    uhub5: <vendor 5="" 9="" 0x0557="" product="" 0x8021,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
    Root mount waiting for: usbus1
    uhub5: 4 ports with 4 removable, self powered
    ugen1.6: <aten international="" co.="" ltd="">at usbus1
    ukbd1: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
    kbd3 at ukbd1
    uhid0: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
    ums2: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
    ums2: 5 buttons and [XYZ] coordinates ID=0
    ums3: <aten 0="" 6="" international="" co.="" ltd="" cs1716a="" v1.3.126,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus1
    Trying to mount root from ufs:/dev/da0s1a
    igb6: link state changed to UP
    pflog0: promiscuous mode disabled
    pflog0: promiscuous mode enabled
    pid 18459 (ntpd), uid 0: exited on signal 11 (core dumped)
    igb6: link state changed to DOWN
    pid 12274 (squid), uid 62: exited on signal 6
    pflog0: promiscuous mode disabled
    pflog0: promiscuous mode enabled
    [2.1-BETA1][admin@pfsense.localdomain]/root(5):</aten></aten></aten></aten></aten></vendor></vendor></mouse></mouse></keyboard></avocent></seagate></no></no></vendor></vendor></vendor></vendor></plds></intel></intel></intel></intel></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></non-standard></at></keyboard></generic></system></isa></at></high></power></dasp></dasp></dasp></dasp></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></intel(r)></intel(r)></intel(r)></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></dasp></dasp></dasp></dasp></acpi></acpi></dasp></ata></ata></intel></serial></ata></ata></intel></isa></pci-isa></acpi></acpi></ehci></ehci></vga-compatible></acpi></acpi></acpi></acpi></intel(r)></acpi></acpi></intel(r)></intel(r)></acpi></acpi></intel(r)></intel(r)></intel(r)></intel(r)></acpi></acpi></ehci></ehci></simple></simple></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></serial></serial></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></lenovo></software></version></version></version></lenovo></lahf></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    ```</x>


  • Interesting, I can confirm that you should see the mfi kick in during boot but it doesn't.
    (BTW: Could you cut the 'dmesg' and 'ls' output to make yourpost more readable + use code tags http://docs.simplemachines.org/index.php?topic=57.0?) :-)

    I guess that you don't see it doesn't attach to a device seen in the output of 'pciconf -lvb' too.
    I do have a couple of devices that don't have a driver attached in this box, maybe you can see the a device without driver that looks like the Lenovo RAID card?
    Although I'm not into driver hacking, maybe what you're missing is the PCI ID for this board, if you can identify the card in pciconf output, trying to add the ID to the driver code could possibly fix things.

    Just guessing: In contrast to IBM which must have taken the same ID as LSI, Lenovo may have taken another PCI ID for their card, the driver doesn't attach because it's unknown.
    (cross-flashing to LSI or IBM firmware could be an option - this is not covered by warranty - but largely covered on the net for ZFS builds)

    I've once read about an issue that when mfi doesn't kick in, mpt or mps, the drivers for non-RAID HBAs take over (which most often does not work) -
    this could explain why the OS may seem to see the single disks instead RAID volumes.

    Here is an example for an unknown device and the M1115 which for the mfi(4) does look identical to its LSI counterpart for the OS:

    
    # pciconf -lvb
     <snip!>none0@pci0:0:4:0:	class=0x088000 card=0x3c201014 chip=0x3c208086 rev=0x07 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = 'Sandy Bridge DMA Channel 0'
        class      = base peripheral
        bar   [10] = type Memory, range 64, base 0xfbf1c000, size 16384, enabled
     <snip!>mfi0@pci0:27:0:0:	class=0x010400 card=0x040d1014 chip=0x00731000 rev=0x03 hdr=0x00
        vendor     = 'LSI Logic / Symbios Logic'
        device     = 'MegaRAID SAS 9240'
        class      = mass storage
        subclass   = RAID
        bar   [10] = type I/O Port, range 32, base 0x3000, size 256, enabled
        bar   [14] = type Memory, range 64, base 0x91d40000, size 16384, enabled
        bar   [1c] = type Memory, range 64, base 0x91d00000, size 262144, enabled</snip!></snip!> 
    

    Although I did investigate a lot on my own, I was very happy to get the assistance of the BSDPerimeter crew for for the crucial parts.
    (without making a shameless plug: I was very thankful for their inputs, if you are doing this for a business, do consider a subscription…) :-)



  • Matsim,

    I have found the root of the problem here after running pciconf. I made a bad assumption. This is not a LSI MegaRAID SAS 9240-8i!

    It is an embedded Intel SAS Controller with an LSI MegaRAID software RAID riding on top of it.

    Lenovo calls it its "RAID 300" interface.

    The Vendor and Device IDs are

    0x8086  0x1d68  <–- Intel C606 Patsburg Storage Control Unit

    The isci driver (Intel) is in control of the device, not mfi.  The isci driver doesn't understand the LSI Software RAID and essentially bypasses it.

    Lesson learned:  use the pfsense HCL!



  • Crap, yes this seems to be one of those "shabby" Intel Chipset-integrated things that can run in a Intel or with LSI or even Adaptec firmware mode depending on the OEM.
    SuperMicro also makes a couple of boards that have this C606 PCH-integrated - somehow "FakeRAID" stuff IMMV. More expensive boards from them  actually have dedicated LSI chip instead of the onboard PCH SAS.

    Anyhow, you're not alone also FreeNAS people have been hurt by this: http://forums.freenas.org/archive/index.php/t-8488.html

    Since you have the RAID 300 thing, this one definitely is onboard (according to Lenovo RD430 manuals)

    Now I'm pretty sure that the RAID 500 would have been the the card you would be looking for and is considered the 9240 card from Lenovo.
    This kind of cards are quite affordable, actually the IBM M1015 is quite popular among in DYI ZFS (FreeNAS) builds.  Maybe you want to give it a
    try, either via the official RAID 500 board or the IBM one. The original LSI cards often are more expensive - but in case of the 9240 come with RAID5 mode unlocked.

    The question actually yet is if you want to fight with the PCH-integrated SAS - actually a vanilla FreeBSD 8.3 or better 9.1 could be worth a try to compare - maybe you can
    kldload isci over there. OR you go the route and shell out some money for a more "proper" SAS RAID card. Actually I'd even prefer a plain HBA in some situations, but some
    server makers like IBM and Lenovo only support and sell you SAS Hardware RAID cards.

    Edit: See http://svnweb.freebsd.org/base/release/8.3.0/sys/dev/isci/isci.c?revision=234063&view=markup , the 0x1d68 device should be handled by iscsi, thus trying with a plain FreeBSD 8.3 could be interesting, but YMMV if you want to spend the time with this card or get a true MegaRAID card :-)


Locked