Using a hard-disk in a Watchguard Firebox X750e for cache/log storage
-
Nice write-up, however I think I'm with ghostshell here. What is the advantage of doing this over running a full install booting from the hd?
Steve
-
CF is more reliable for read-only access and a rotating-rust hard disk will last much longer than an SSD where a lot of writes are being performed. This combo therefore makes (cheap) sense for a proxy cache.
One downside is speed… I'm only getting about 4.5Mbytes/s IO on the hard-disk which is no faster than my broadband connection.
Has anybody tested SSD speed using diskinfo?
The following is from a 40G 5400rpm drive, a Seagate ST9402115A:
diskinfo -tv /dev/ad1 /dev/ad1 512 # sectorsize 40007761920 # mediasize in bytes (37G) 78140160 # mediasize in sectors 0 # stripesize 0 # stripeoffset 77520 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. 5PV09ZED # Disk ident. Seek times: Full stroke: 250 iter in 8.014669 sec = 32.059 msec Half stroke: 250 iter in 6.244615 sec = 24.978 msec Quarter stroke: 500 iter in 9.976294 sec = 19.953 msec Short forward: 400 iter in 3.585714 sec = 8.964 msec Short backward: 400 iter in 2.533780 sec = 6.334 msec Seq outer: 2048 iter in 0.433188 sec = 0.212 msec Seq inner: 2048 iter in 0.384784 sec = 0.188 msec Transfer rates: outside: 102400 kbytes in 22.705212 sec = 4510 kbytes/sec middle: 102400 kbytes in 22.845457 sec = 4482 kbytes/sec inside: 102400 kbytes in 22.782483 sec = 4495 kbytes/secSteve
-
But what happens if the disk fails? Will the box still boot? If not then you'd be better off just booting from the HD. At least then you can update the firmwhere without having to modify the scripts again.
Steve
-
Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure. If the disk fails, just pull it from the rear of the box, then pfSense can be up and running while you wait for a new disk to arrive off eBay, plug in the new one and carry on, albeit having lost some logs and the (not very valuable) cached data which is probably stale by that point anyway.
If fact, even if somebody tells me that an SSD is fast by comparison, I'll probably still stick with the CF for the OS, and the SSD for /var. Now that I have this working, I'm considering a 32G SSD drive as they're cheap, and spread over that size, the amount of writing my pfSense installation will do will probably see it last well.
Steve
-
Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure.
Ok, you've clearly thought this through.
Personally the ability to upgrade the firmware without having to re-edit the scripts outweighs the advantages offered by running from flash. That's just my situation though.Steve
-
Hi Steve,
What scripts are you referring to?
I run bacula on my home network, and my pfsense box has been added to that regime, so I could perform an upgrade and then restore modified scripts if desired.
Steve
-
What scripts are you referring to?
/etc/rc.embedded
If you can restore it easily enough then I guess it's less of an issue. Having more options is always good.
Steve
-
Thanks to a post HERE by stephenw10 I've realised that my hard disk didn't have DMA enabled.
[2.1-RELEASE][admin@pfsense]/root(18): atacontrol mode ad1 current mode = PIO4 [2.1-RELEASE][admin@pfsense]/root(19): atacontrol mode ad1 UDMA6 current mode = UDMA33 [2.1-RELEASE][admin@pfsense]/root(20): diskinfo -tv /dev/ad1 /dev/ad1 512 # sectorsize 40007761920 # mediasize in bytes (37G) 78140160 # mediasize in sectors 0 # stripesize 0 # stripeoffset 77520 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. 5PV09ZED # Disk ident. Seek times: Full stroke: 250 iter in 7.989798 sec = 31.959 msec Half stroke: 250 iter in 6.254760 sec = 25.019 msec Quarter stroke: 500 iter in 9.845523 sec = 19.691 msec Short forward: 400 iter in 3.577064 sec = 8.943 msec Short backward: 400 iter in 2.471400 sec = 6.179 msec Seq outer: 2048 iter in 0.187117 sec = 0.091 msec Seq inner: 2048 iter in 0.332214 sec = 0.162 msec Transfer rates: outside: 102400 kbytes in 3.476458 sec = 29455 kbytes/sec middle: 102400 kbytes in 3.863341 sec = 26506 kbytes/sec inside: 102400 kbytes in 5.491517 sec = 18647 kbytes/secUp to six times the speed. I'm happy with that!
Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.
Steve
-
Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.
Steve
Setting these sysctl variables, in loader.conf.local, didn't enable DMA?
hw.ata.wc="1"
hw.ata.atapi_dma="1"
hw.ata.ata_dma="1" -
Doing that enables DMA globally which includes the CF card. Since the CF-IDE adapter in the firebox doesn't support DMA it will fail to boot with pages of DMA errors.
Looks like the Shellcmd package is the way to go, no way of selectively enabling DMA per device.
Have you tried any BIOS DMA settings?Steve
-
Hi Steve,
Indeed, the issue here was that I wanted to selectively enable DMA on the hard disk only.
No, I've not looked at the BIOS. With a current uptime of 50 days I'm loathed to reboot it to be honest. What BIOS settings are there?
Steve
-
There are options for dma mode for primary master and slave separately. However those options were originally hidden and they are both set to UDMA disabled by default. Since udma still seems to be available maybe those options are not implemented. Easy test though.
Steve
-
Now I followed the post (which is exactly what I wanted to do as well) but for whatever reason, I have two /var still mounted. I have rebooted it a couple of times to see if I can see where it occurs or why, but have not been able to figure it out. Any suggestions I can check as to why? Otherwise, it works perfectly. df -h is below:
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/pfsense0 442M 379M 28M 93% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ufs/cf 49M 1.4M 44M 3% /cf
/dev/ad1s1a 36G 10k 33G 0% /var
/dev/md0 38M 90k 35M 0% /tmp
/dev/md1 57M 17M 35M 33% /var
devfs 1.0k 1.0k 0B 100% /var/dhcpd/devThanks for any help!
-
What does your modified rc.embedded look like?
You should see the various messages from the modified script in the boot log, are you seeing those? Are they correct?Steve
-
The memory device /dev/md1 is still being created/mounted so as Stephen says you've got an issue with the startup script.
Steve
-
No. I see now, that it is incorrect. I still have the original being mounted due to it. My question is, where specifically in the original rc.embedded do I change with the Steve's changes as there are other calls in the script that I am not sure are needed or not. Apologies, just new to the pfsense and Watchguard platforms so I am sure my questions are on the the dumber side…. The original rc.embedded is below:
rc.embedded - embedded system specific startup information
For pfSense
Size of /tmp
USE_MFS_TMP_SIZE=
/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then
tmpsize="${USE_MFS_TMP_SIZE}m"
else
tmpsize="40m"
fiSize of /var
USE_MFS_VAR_SIZE=
/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/ cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'
if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then
varsize="${USE_MFS_VAR_SIZE}m"
else
varsize="60m"
fiRun some initialization routines
[ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start
echo -n "Setting up memory disks…"
mdmfs -S -M -s ${tmpsize} md /tmp
mdmfs -S -M -s ${varsize} md /varCreate some needed directories
/bin/mkdir -p /var/db
Ensure vi's recover directory is present
/bin/mkdir -p /var/tmp/vi.recover/
echo " done." -
Here is the complete patched /etc/rc.embedded.
#!/bin/sh # # rc.embedded - embedded system specific startup information # For pfSense # Size of /tmp USE_MFS_TMP_SIZE=`/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'` if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then tmpsize="${USE_MFS_TMP_SIZE}m" else tmpsize="40m" fi # Size of /var USE_MFS_VAR_SIZE=`/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'` if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then varsize="${USE_MFS_VAR_SIZE}m" else varsize="60m" fi # Run some initialization routines [ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start echo -n "Setting up memory disks..." mdmfs -S -M -s ${tmpsize} md /tmp # If a hard disk is installed then mount that on /var # otherwise use a ramdisk harddisk="/dev/ad1s1a" if [ -c $harddisk ] then echo -n "Using /var physical disk..." mount -o noatime $harddisk /var # Ensure /var/run is removed on boot before daemons are started # It should arguably be a tmpfs, but this works fine rm -r /var/run # sshd won't start if /var/empty exists on boot rm -r /var/empty else echo -n "Using /var memory disk..." mdmfs -S -M -s ${varsize} md /var # Create some needed directories /bin/mkdir -p /var/db # Ensure vi's recover directory is present /bin/mkdir -p /var/tmp/vi.recover/ fi echo " done." -
Thank you so much Steve! Works perfectly. Thanks for your patience….
-
only my result of a x5500e with a ide SSD from CWC :
/dev/ad1s1a
512 # sectorsize
57872344064 # mediasize in bytes (53G)
113031922 # mediasize in sectors
0 # stripesize
40448 # stripeoffset
112134 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
OW140822AS1514144 # Disk ident.Seek times:
Full stroke: 250 iter in 0.040570 sec = 0.162 msec
Half stroke: 250 iter in 0.037379 sec = 0.150 msec
Quarter stroke: 500 iter in 0.125276 sec = 0.251 msec
Short forward: 400 iter in 0.093555 sec = 0.234 msec
Short backward: 400 iter in 0.096201 sec = 0.241 msec
Seq outer: 2048 iter in 0.169674 sec = 0.083 msec
Seq inner: 2048 iter in 0.123082 sec = 0.060 msec
Transfer rates:
outside: 102400 kbytes in 1.124830 sec = 91036 kbytes/sec
middle: 102400 kbytes in 1.128009 sec = 90779 kbytes/sec
inside: 102400 kbytes in 1.125042 sec = 91019 kbytes/secseems that i don't have to tune anything.
I will make the same test with a x750e soon, will let you know if somebody is interrested :)
Regards -
What tool are you testing that with? Did you enable TRIM? Be interesting to see how those figures vary over time once you've moved a few gigs across the drive.
Steve
