SSD (Solid State Drive) and pfSense (Important)

stephenw10 · Mar 13, 2011, 10:49 PM

If you are testing this with a Nano installation you should be aware that there is currently a bug in 2.0rc1 which means that the filesystem is left RW after boot. It should be RO.
However it does not cause any undue writes as everything is set to run from RAM as you say.
If you check the filesystem don't be alarmed to find it's still RW, as I was! ;)

Steve

Edit: Bug listed here.

FJSchrankJr · Mar 14, 2011, 3:11 AM

@stephenw10:

If you are testing this with a Nano installation you should be aware that there is currently a bug in 2.0rc1 which means that the filesystem is left RW after boot. It should be RO.
However it does not cause any undue writes as everything is set to run from RAM as you say.
If you check the filesystem don't be alarmed to find it's still RW, as I was! ;)

Steve

Edit: Bug listed here.

Hey Steve – thanks for the heads up! I did install pfSense back to the SSD. I went the embedded route and I took my 8GB SSD drive and loaded the 4GB nano image (didn't really need the extra space). It booted and loaded in to ram and solved my problem. Everything including RRD, /tmp and logs all run from ram now. According to stats the disk writes are virtually nothing except for config file changes. Only downside to running on embedded is you lose the VGA/Keyboard console and have to go serial to configure but not really an issue. Who knows maybe going this route will be more reliable because VGA isnt being loaded.

This looks like the perfect solution for SSDs (the same setup as CF). pfSense really did take these issues in to account when they created the embedded/CF version.

Thanks for the help everyone.

dreamslacker · Mar 14, 2011, 4:29 AM

@FJSchrankJr:

Just checked on some things and the nanobsd version may not be a concern. It looks like you're right and the file system is loaded in to memory. Someone can hopefully confirm this for you but I think you are ok.

The nanobsd version can be configured to write out the logs/ RRD data at specific intervals like 30 minutes or 1 hour etc. This allows you to retain information across reboots and yet heavily reduce the amount of I/O to the SSD/ CF.

Next on the point of the SSD, most controllers are optimized for Windows (unfortunately). They rely on the Trim command to do garbage collection.
However, the Indilinx based drives (OCZ Vertex, Corsair X128 etc) have hardware level garbage collection which isn't as efficient but beats having none under *nix. This would be useful to improve general performance in the long run.

The Sandforce based drives offer very very impressive write amplification and compression algorithms. This would be good for wear levelling and long term reliability. A write amplification of 0.5X means that The equivalent wear and tear in the long run is half the data being written. This is obtained through clever compression tricks and since the RRD/ log data is easily compressible, this is very effective as well.
SSDs like the OCZ Vertex 2(e), Gskill Phoenix Pro use the the Sandforce controllers.

I did a short write-up on the OCZ Vertex 2E drive here:
http://vr-zone.com/articles/old-dog-new-tricks-ocz-vertex-2e-reviewed/10323.html

The first section lists out the very basic pros of using the Sandforce controller and how it helps reduce the amount of write penalty in terms of performance and wear and tear as well.
Also in the testing is the Indilinx based drive which will show the difference between the effectiveness of garbage collection algorithms between the 2 controllers and the performance differences as well.

Of note is the difference between the compressible and incompressible data testing. If you're running embedded with mostly compressible data written (logs and such), the Sandforce is likely to be a better choice.
If you run Squid and most of the content being cached are large compressed EXE files or video files, then the Indilinx is likely to be your best bet.

koukobin · Mar 14, 2011, 5:45 AM

If you want to have VGA and console output at the same time, you can use Hacom nanobsd Pfsense images:

http://www.hacom.net/catalog/pub/pfsense

lowemissions · Mar 14, 2011, 9:15 PM

SSD?
pfsense system spec:
wyse 3455xl
Ram 256 SDRAM
HD: 4 gig SSD IDE DOM
NIC: 4 Ethernet (intel) dr0-WAN, dr1-LAN, dr2-OPT1, dr3-OPT2

I installed on a 4 gig DOM SSD by Live CD for about 10 days now. I saw FJSchrankJr post a warning about SSD. What is the best solution for me? Can I load the nanobsd image by the Gui to fix my problem? or I have to reinstall from clean install.

Thank

CNLiberal · Mar 14, 2011, 9:24 PM

@dreamslacker:

The nanobsd version can be configured to write out the logs/ RRD data at specific intervals like 30 minutes or 1 hour etc. This allows you to retain information across reboots and yet heavily reduce the amount of I/O to the SSD/ CF.

Can you explain this process (or link to the page)? I just picked up a 4GB CF card to run my pfSense 2.0 install from and it'd be great to keep the graphs. Thanks!

Jim

stephenw10 · Mar 14, 2011, 9:29 PM

@lowemissions:

What is the best solution for me? Can I load the nanobsd image by the Gui to fix my problem? or I have to reinstall from clean install.

You will want to switch to either the embedded install, sellected when you install from CD, or a NanoBSD image. You will want to do it quickly before your DOM dies! ;)

You can probably backup your config file and restore it to the fresh install. You can't switch the install type from the GUI.

Steve

lowemissions · Mar 14, 2011, 9:47 PM

Thank Steve

tester_02 · Mar 15, 2011, 12:33 AM

FYI I've had 1.23 full install running on a 40gb intel ssd (spare at time)for at least 1.5years so far. No issues.

stephenw10 · Mar 15, 2011, 1:20 AM

Do you have a lot of ram in that machine, enough to prevent swapping?

With a 40GB drive I imagine you will have a lot of empty space, the ware leveling on the drive will swap data around such that 10,000 writes can go along way.

Even so I'd keep checking the SMART status of the drive if I were you.

Steve

cmb · Mar 15, 2011, 5:24 AM

Note the logs are kept in RAM whether you're running a full install or nanobsd.

A lot of people are running normal full installs on SSD. Sure they don't have infinite writes, but it's enough it should on average last as long as a typical hard drive. It's unlikely you're touching swap at all on your firewall, if you are you have serious issues.

@CNLiberal:

Can you explain this process (or link to the page)?

Check Diagnostics>Nanobsd, the settings for the periodic automatic copy to CF of RRD data are there.

stephenw10 · Mar 15, 2011, 12:27 PM

@cmb:

A lot of people are running normal full installs on SSD. Sure they don't have infinite writes, but it's enough it should on average last as long as a typical hard drive. It's unlikely you're touching swap at all on your firewall, if you are you have serious issues.

Hmmm, that's interesting. I assume you are talking about larger, HDD replacement type drives such as tester_02's 40GB?

I did some research on this a while ago and came to the conclusion that it just wasn't worth the extra investment in a firewall appliance. The speed increase provided by a SSD is unlikely to be of much benefit except in boot up time which doesn't count for much.
I suppose there are enough people running a Windows install from SSD that we'd be seeing failures by now if it was a problem.

I'd be interested to hear the reasons behind running an SSD from anyone doing so.

Steve

FJSchrankJr · Mar 18, 2011, 3:27 PM

@lowemissions:

SSD?
pfsense system spec:
wyse 3455xl
Ram 256 SDRAM
HD: 4 gig SSD IDE DOM
NIC: 4 Ethernet (intel) dr0-WAN, dr1-LAN, dr2-OPT1, dr3-OPT2

I installed on a 4 gig DOM SSD by Live CD for about 10 days now. I saw FJSchrankJr post a warning about SSD. What is the best solution for me? Can I load the nanobsd image by the Gui to fix my problem? or I have to reinstall from clean install.

Thank

Hi – if you are running off of the LiveCD then everything is mounted in memory. However, if you install that the file system will be loaded to your hard drive.

The only way that I know of to switch to nanobsd version is to do a clean install like I did. I should warn you that once you load NanoBSD version everything except the config files will be running in RAM and 256 might not be enough. In the past I had been running on 256 and a hard drive install with no problem but when I switched to NanoBSD version (upgraded to 512Mb RAM) I did run out of memory. I am not sure if it was some type of memory leak or cache but I will have to install more memory because the system almost crashed.

The easiest way to upgrade is to make a config backup, write the NanoBSD image to the drive and boot. If your main firewall cannot go down then try to find another machine to act as a mirror. That's how I did it, then just restored the config and I was up and running. Good luck!

I did notice on pfSense 1.2.3 the Embedded option during the LiveCD install was not the NanoBSD version but just the disabled vga version. Make sure you use the NanoBSD version from the download mirrors.

CORRECTION: 256Mb is more than enough RAM for a base embedded install. However, adding additional packages, logging, etc. where memory usage will be higher consider using memory of 512Mb+.

FJSchrankJr · Mar 15, 2011, 12:55 PM

@tester_02:

FYI I've had 1.23 full install running on a 40gb intel ssd (spare at time)for at least 1.5years so far. No issues.

Some SSD drives use hard drive controllers that spread wear across the entire drive. 40GB drive would give the controller plenty of space to spread. If you are not running NanoBSD or an optimized regular install, you will eventually crash. SSDs and Compact flash should be treated the same way. One of the main reasons for the NanoBSD version was to address the CF write limitations.

stephenw10 · Mar 15, 2011, 12:56 PM

256MB should be no problem, as long as you don't have a memory leak! :P

That's the standard amount in the Watchguard X-Core and there are many people running that with NanoBSD.

Steve

FJSchrankJr · Mar 15, 2011, 1:03 PM

@stephenw10:

256MB should be no problem, as long as you don't have a memory leak! :P

That's the standard amount in the Watchguard X-Core and there are many people running that with NanoBSD.

Steve

Hi Steve, I do agree! I am running on pfSense 2.0 (NanoBSD) with 512Mb of RAM and no packages installed. Overnight I went to about 98% memory and DHCP server crashed. All logging is disabled so either there is a cache building somewhere in ramdisk or I do have a leak.

FJSchrankJr · Mar 15, 2011, 3:05 PM

@FJSchrankJr:

@stephenw10:

256MB should be no problem, as long as you don't have a memory leak! :P

That's the standard amount in the Watchguard X-Core and there are many people running that with NanoBSD.

Steve

Hi Steve, I do agree! I am running on pfSense 2.0 (NanoBSD) with 512Mb of RAM and no packages installed. Overnight I went to about 98% memory and DHCP server crashed. All logging is disabled so either there is a cache building somewhere in ramdisk or I do have a leak.

Good news is that my memory issue on 512Mb is not a leak. I guess we're pushing more traffic then I thought because as the MBUFs increase dynamically, the memory is jumping up too. This was not an issue on the non NanoBSD version because the file system was on disk and gave me much more RAM.

The NanoBSD version really is the perfect solution running on a SSD or CF drive. I am not running the traffic shaper on this system but if you run the traffic shaper make sure you have a lot of memory. The queues need lots of memory.

If you put pfSense on the right hardware, run NanoBSD, use a SSD, moderately high-speed CPU and a lot of memory pfSense performance is terrific and is rock solid.

cmb · Mar 16, 2011, 5:56 AM

@stephenw10:

@cmb:

A lot of people are running normal full installs on SSD. Sure they don't have infinite writes, but it's enough it should on average last as long as a typical hard drive. It's unlikely you're touching swap at all on your firewall, if you are you have serious issues.

Hmmm, that's interesting. I assume you are talking about larger, HDD replacement type drives such as tester_02's 40GB?

Yeah, 30+ GB.

In general, it's not worth it for a firewall in most environments. Unless you have something like Squid installed, the disk is almost never touched after you're booted up anyway.

FJSchrankJr · Mar 16, 2011, 4:00 PM

mb link=topic=34381.msg178857#msg178857 date=1300254998]
@stephenw10:

@cmb:

A lot of people are running normal full installs on SSD. Sure they don't have infinite writes, but it's enough it should on average last as long as a typical hard drive. It's unlikely you're touching swap at all on your firewall, if you are you have serious issues.

Hmmm, that's interesting. I assume you are talking about larger, HDD replacement type drives such as tester_02's 40GB?

Yeah, 30+ GB.

In general, it's not worth it for a firewall in most environments. Unless you have something like Squid installed, the disk is almost never touched after you're booted up anyway.

Hi cmb: In the NanoBSD version after boot the disk is not touched but if you are running the normal install and have not disabled log writing, RRD graphing, or any other write intensive service the SSD will be destroyed and quite possibly long before a typical hard drive. 10k writes per cell is nothing and if RRD and logs are constantly writing with each write it's destroying the NAND cells. If you're using a 30GB SSD and the memory controller in the SSD utilizes wear leveling then it has plenty of space to remap/spread across. However, if numerous writes are occurring constantly then it's a matter of time before it does fail, 30GB gives you more time. Also Hybrid SSDs are different because of the RAM cache, those would work a bit longer. Just use NanoBSD version on a SSD and it will last a very long time.

Wear leveling: http://en.wikipedia.org/wiki/Wear_leveling

SSDs are fairly new technologies and are not ready for server environments with frequent writes so it's very important to treat the current SSD technology the same way you do as CF. Embedded pfSense (NanoBSD) version is perfect for SSDs. SSDs are not a simple drop-in replacement on the normal pfSense install and if you use the embedded/NanoBSD version no additional steps are needed.

When running off of the LiveCD it does mount in memory but once installed the file system is loaded on the drive and not in RAM. There is a RAM disk but it is not used for the /tmp directory (used for RRD storage amonst other stuff) on the normal pfSense version. On embedded/NanoBSD version it runs even the /tmp directory in RAM.

You're right about the swap file, typically this will never be used unless you run out of memory. However the regular version of pfSense is just not optimized for CF/SSD memory and so disk writes will be occurring unless you disable the other services I mentioned in the first post.

stephenw10 · Mar 16, 2011, 4:35 PM

I think you may be underestimating the number of cells in a 30GB drive available for wear leveling.
If it were true that SSDs would wear out rapidly wouldn't we be seeing more failures among all the SSD netbooks and macbook airs?

Consider that Intel said, upon launching their 80GB X-25:

Our MLC SSD can allow a user to write 100GB/day every day for five years without wearing out the drive

However that doesn't tie in with the second post in this thread. 16GB drive, 2.5 months, dead!

I suspect that it's very dependent on the algorithms in the drive controller.

Then again I ran Windows 98 from a 128MB CF card for a few years with no problems! ::) (though I did disable swap)

Steve