Boot from USB, and use hdd for cache/temp files?

mickeyholland · Apr 13, 2012, 10:25 AM

Hi all,

I was thinking about power saving and resilience for a new installation, which might be on an HP Microserver N36L.

That server can boot from USB, and comes with a 250Gb sata drive for a very good price :)

I could make a couple of usb stick images of my pfsense install including the config for my network (so one usb as 'cold-swap' replacement if first failed), and boot and run pfsense from usb. However, I would like to use some caching and logging, so could that temporary data be written to the hdd as I understand that might destroy the usb fairly quickly with lots of writes?

If not, I would perhaps need to buy a second hdd, configure as raid1, then ditch the usb stick idea. But that means extra cost for the hdd, and extra running costs (electric bill). That is why I was thinking about the usb/hdd combo.

Has anyone done this… is it a reasonable or silly idea?

Thanks!!

stephenw10 · Apr 13, 2012, 2:10 PM

It can and has been done.
Because it's not a standard install it probably wouldn't survive a firmware update. Something to bare in mind.
Search the forum for: 'squid cache HD nanobsd' or some similar combination.

E.g. http://forum.pfsense.org/index.php/topic,40780.0.html

Steve

mickeyholland · Apr 13, 2012, 2:30 PM

Thanks for the reply.

Actually… posting here helped another idea click...

If I am using a single hdd for cache, then I might as well install pfSense to that single hdd, and when all configured make a clone of the hdd to an identical spare (perhaps with dd if pfsense has that?).

If the hdd should fail in the future, just swap drives. It means only 1 drive is running (burning electric), and I save the complication of usb sticks! I guess I lose any logs, dhcp lease info or cache if the hdd fails, but probably they are not important and I would have lost those anyhow with the usb-stick scheme!

stephenw10 · Apr 13, 2012, 8:17 PM

That will definitely be easier. :)
There are some instruction in the wiki if you change your mind:
http://doc.pfsense.org/index.php/Local_Disk_Storage_on_Embedded_%28soekris%29

Though as I said in another thread one of the main reasons for booting from flash is the reliability of solid state storage. Introducing a HD reduces this. Even if you continue to boot from flash do you know what the box will do if the HD fails whilst in use as a squid cache? If the answer is squid crashes and http connectivity is lost then you might as well have been booting from the HD anyway.

Steve

robi · Apr 13, 2012, 8:59 PM

@mickeyholland:

…as I understand that might destroy the usb fairly quickly with lots of writes...

True, that depends on how many writes there are.

A small install for a home network on my SanDisk USB stick killed it after about 2 years of non-stop operation. I wouldn't call that fairly quickly, but your mileage may vary.

Now I use NanoBSD on CF disk, emtirely read-only partition ;D Really cool feature.

robi · Apr 13, 2012, 9:09 PM

As I see the HP Microserver N36L has Embedded AMD SATA controller with RAID 0, 1. Why not make a mirrored RAID 1 setup with two disks? If one dies, you're still alive with the other one. And you can change disk on the fly by hotplugging. Just make sure you buy a different brand and model disk of the same szie. Or even if you're more paranoid, you can make RAID 1 with 4 disks - super safe.

mickeyholland · Apr 14, 2012, 7:20 AM

Hi Robi! Thanks for sharing!

Just one thing to be careful… I am not certain the standard HP Microserver allows hot-swap!
In the specs it describes the storage capability as: "Non-Hot Plug SATA"

Anyhow my desire is to have just 1 drive (or less) running to meet certain low-power requirements. That is why I would not consider RAID1 in this case.

As general theory… And for a server which is not critical (can afford 30 minutes downtime 2 or 3 times a year for regular cleaning and perhaps hdd swaps!) and does not have data / configuration which is often changing, then I do not agree RAID is the best way, even if hot-swap was available. It seems pointless to be burning out 2 or more drives 24/7 if not needed for performance reasons. I prefer to configure the system, make a clone hdd, and have it ready to plug in if the worst happens with the primary hdd. Just seems to me a far more eco-friendly approach on many levels!

I still like the idea of CF or USB running the server and using no hdd. I am still a little unsure of what I can expect performance wise! Perhaps you already know the answer?? My approx. system would be:

50 users on LAN (web / email / local file share - certainly not power users!)
dmz with email, www (very light usage, maybe 2-3 visitors/day!)
~300 Mb fiber internet connection (that's what speedtest reports) but the users are mostly limited by 100Mb media converters anyhow!
pfsense should provide: dhcp, dns, bandwidthd, snort, havp-av, pfBlocker, squid, squidguard.

Is it realistic to run this from usb or CF? Or really should I use hdd install. Also, what throughput might I expect from the N36L server. Is that gonna be a limiting factor? I noticed in a previous post stephenw10 mentioned the D525 can push ~400Mbs, and this processor scores better than that in the cpubenchmarks (N36L=~800, D525=~700).

The nics are Intel PRO/1000 MT btw!

stephenw10 · Apr 14, 2012, 8:14 AM

Clearly the earlier me was misinformed! (though I guess it varies by other factors) ::) The D525 can push greater than 600Mbps of firwall/NAT traffic so you should be in that area. However that's without snort, squid, havp etc. Each of those will slow you down quite a bit.

There is little difference in performance terms between the NanoBSD install and full HD install.

If you need to run squid for caching purposes then you need a HD.

Steve

mickeyholland · Apr 14, 2012, 9:54 AM

Should I read that as; if I drop the squid & squidguard from the list of duties, then pfsense might perform equally well with usb-nanobsd install (compared to hdd install) for many (well at least 2) years before a reasonable quality usb 2.0 might fail from writes-limit.

If performance will be much the same, and I get 2+ years per usb stick, then I am leaning back that way as it seems of great heat/electric benefit not to run any hdd.

I guess the (daily?) virus updates would be written to the usb stick, but once per day should not be a problem for usb stick- if I choose a 2gb stick, then I must have 5-10 years life easily? Not sure what else would be written (Except the config of course, but only when I change something?)

stephenw10 · Apr 14, 2012, 10:20 AM

The actual performance of the system will not vary much between Nano and HD installs, once its has booted its runs almost entirely from ram anyway. A full install has swap space on the drive which can start to slow things down but if you have enough memory that should never be a problem. Of course Nano doesn't help that it just runs out of memory which causes other issues.
NanoBSD is specifically designed for running from flash memory. It mounts the file system read only and has no swap as such it should never fail due to write limits. Robi's two year experience was, I think, running a standard install from a flash drive which is not recommended, at all! YMMV ;).
However because it has a read only file system it is more limited in which additional packages can be installed. When you open the package installer in the webGUI it shows only packages that can be installed on your platform. It's somewhat tedious but you can look through the package list manually here:
http://www.pfsense.org/packages/pkg_config.8.xml
Anything marked <noembedded>true</noembedded> cannot be used with a NanoBSD install. Other packages can be used but are restricted, Squid for example.

It's pretty easy installing Nano to a USB drive so you could just try it. However all things considered I would probably go with a full install given your requirements.

Steve

mickeyholland · Apr 14, 2012, 12:53 PM

Thanks for all the great advice. I will give it a try, if not for anything else than should be a great learning experience.

What sort of system ram should I put in? 1gb enough, or rather more? If I rely on that for the ram-swap, maybe I need a lot if I try to install several packages… is there a rough-rule for calculating minimum ram with nano installs?

stephenw10 · Apr 14, 2012, 5:01 PM

The absolute minimum requirement is 128MB, there are plenty of systems running 256MB. I have 512MB in my main box but I don't run any memory hungry packages.
However if you want to run Snort then you can never have too much!
If I remember right those microservers have lots of ram slots? So you can always add more if 1GB is not enough. Ram is so cheap these days it's often more expensive to try to get small amounts.

Steve

mickeyholland · Apr 14, 2012, 7:01 PM

OK, understood.

Thank you again. It is great to get a little knowledge comfort before ordering and playing! This forum is great!

I will post my results here in about 2 weeks in case anyone else can benefit. Just need to order the new server first!
:)

robi · Apr 20, 2012, 4:41 PM

@stephenw10:

Robi's two year experience was, I think, running a standard install from a flash drive which is not recommended, at all!

That actually was not a pfSense installation. It was an Optware package system added to a DD-WRT router, extending the capabilities of an Asus WL-500GP router with standard DD-WRT firmware to lots of clever things (including Asterisk).

Dropped that setup mainly because it wasn't able to handle the increasing WAN bandwidth available (Asus WL-500GPv1 has a 266 MIPS CPU with 32MB of RAM and 8MB of NAND Flash).

Now I'm planning to re-use these Asus routers with DD-WRT-based Linux firmware as OpenVPN clients to pfSense, on smaller remote sites. For this I won't need any USB sticks at all, OpenVPN binary is already compiled in the fw, and there's about 500kB of free space in the NAND which can be mounted as JFFS partition to hold custom configs. Disabling logging and it will run just fine for a couple of years…