NVMe SSD installation fails, won't install to drive despite finding it.

James5mith · Jan 14, 2016, 3:28 PM

I'm hoping this is PEBKAC and not an actual bug/issue.

Seems that the default installer in 2.2.6 64-bit refuses to install to the NVMe drive in my new pfsense box. I can confirm that the system sees the drive, that it can read/write from the drive (when mounted within the shell from a USB stick), and that the NVMe driver is in the kernel.

I have done a FreeBSD 10.2 install on the same system, and it installs to the NVMe drive just fine.

I'm hoping someone here can offer me some ideas on what I might have done wrong, or if there is a hidden option in the installer to use NVMe storage as the primary drive.

Basics of hardware:
Supermicro X10SDV-8C-TLN4F (Xeon D integrated onboard CPU)
32GB DDR4 RAM
Samsung 950 Pro 256GB NVMe SSD

BIOS Settings:
LEGACY Boot #Seems to require this to see the ISO mounted from IPMI unfortunately.
LEGACY M.2. Socket mode
LEGACY PCIe x16 slot mode

*EDIT: As a backup, I have ordered some SATA DOMs in case this is simply something that pfsense cannot do.

bluepr0 · Jan 18, 2016, 11:55 PM

Have you tried 2.3 beta? It's using 10-2 release, so maybe if fixes some weird bug the installer has?

Perforado · Jan 21, 2016, 4:37 PM

Could be a naming issue.

What's the nvme-Device called within FreeBSD?

daX
adaX

which are supported. Or maybe

nvmX

which may not be recognized by the Installer?

Guest · Jan 21, 2016, 9:27 PM

Basics of hardware:
Supermicro X10SDV-8C-TLN4F (Xeon D integrated onboard CPU)
32GB DDR4 RAM
Samsung 950 Pro 256GB NVMe SSD

Wow! This might be a real pfSense bomb. Could you write a small review over that board over perhaps
IPSec VPN and other throughput orientated things?

It must be loaded a proper NVMe driver, so it could be that bluepr0 is right with the 2.3 version!

LEGACY M.2. Socket mode
LEGACY PCIe x16 slot mode

Is this perhaps a shared PCie slot and a jumper must be set to use the M.2 SSD.

*EDIT: As a backup, I have ordered some SATA DOMs in case this is simply something that pfsense cannot do.

Might be a good work around, but as I know the Samsung950 Pro is going to show read 2500 MB/s
and write 1500 MB/s. That is enormously fast by using low electric power.

Perforado · Jan 22, 2016, 7:46 AM

The only spare i had for testing was a Samsung XP941 AHCI M.2 which is detected as an ahci-drive as expected:

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <samsung mzhpu512hcgl-00004="" uxm6501q="">ATA-9 SATA 3.x device
ada0: Serial Number S1NDNYAFA00800
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 488386MB (1000215216 512 byte sectors: 16H 63S/T 16383C)

I could install pfsense on it but it didn't boot on a Supermicro X18SDV sadly. Maybe UEFI-Only?

Will dig further for a real nvme m.2 drive :)

But using an ahci m2 drive would solve your installation problem i guess.</samsung>

James5mith · Feb 18, 2016, 7:38 PM

I've ended up running on the SATA DOM's for now. Both pfsense and freeBSD can see the NVMe drives. (As mentioned previously). But FreeBSD will install, but not boot from it.

I'll try the beta, hadn't even thought of that in my scramble to get these live. But as it stands now, I have one working functional system using the SATA DOM, and my secondary (eventual failover) system that I'm still fiddling with/testing/etc.

@Perforado: It shows up as nvme0

I'm thinking that if I'm stuck for now on the SATA DOMs, that I'll go ahead and map some folders or something over to the NVMe drive within the OS. At least I can take advantage of the speed, even if I can't boot from it natively.

*Edit: Just downloaded and tried Beta 2.3 20160217. It finds the NVMe drive (interestingly as nvda0) and allows me to try the install. Turning on swap fails miserably, and there are some other errors. But it definitely seems like the new version of the OS will support NVMe a lot better than the current stable 2.2.6.

Guest · Feb 19, 2016, 1:27 AM

But FreeBSD will install, but not boot from it.

Perhaps it is to new and/or might be not really right implemented in FreeBSD and also pfSense.

Perhaps this three threads might be enlighten you a bit better that NVMe is not really ready until
now (FreeBSD 9.0/11.0).

Partition and file system creation on NVMe based SSD
NVMe performance 4x slower than expected

I really think it will be need more time to get it right working but then it could be a really good chance
to assemble a really fast and power saving appliance.

James5mith · Feb 19, 2016, 2:15 PM

I should note (if it wasn't obvious from my post): pfsense finds the drive, but it does not have the nvd driver loaded properly. (Since I can only see the nvme0 and nvme0ns1 devices, none of the nvdx devices.) That would indicate that by default, pfsense installs the nvme driver, but not the nvd driver that allows for partitioning/formatting.

I'll have to read the links you provided @BlueKobold. Seems like it might not even be worth trying to pivot some of the heavier use folders over to the NVMe for now.

Guest · Feb 19, 2016, 5:46 PM

@James5mith
I was only providing you this two links that will be able to see that in version 9.0 it was not solved and
really ready or working and until now the version 11.0 it is also not really working too. So if pfSense
would change now, or today to the FreeBSD version 11.0 you could also not bot from this device.

Don´t worry they will get this working over a longer or shorter time and then you will be the lucky one. ;)
It might be able to build a real pfSense bomb that is fast like a rocket, the tech specs. from the Samsung950 Pro
is showing something like write 2500 MBit/s and reading 1500 MBit/s.

James5mith · May 4, 2017, 5:39 AM

Gravedigging this to note that it seems that PFSense 2.4 will be the release that is based on FreeBSD11, meaning it should finally be able to install to/boot from NVMe storage at that point.