Using a hard-disk in a Watchguard Firebox X750e for cache/log storage

Steve Evans

CF is more reliable for read-only access and a rotating-rust hard disk will last much longer than an SSD where a lot of writes are being performed. This combo therefore makes (cheap) sense for a proxy cache.

One downside is speed… I'm only getting about 4.5Mbytes/s IO on the hard-disk which is no faster than my broadband connection.

Has anybody tested SSD speed using diskinfo?

The following is from a 40G 5400rpm drive, a Seagate ST9402115A:

diskinfo -tv /dev/ad1
/dev/ad1
	512         	# sectorsize
	40007761920 	# mediasize in bytes (37G)
	78140160    	# mediasize in sectors
	0           	# stripesize
	0           	# stripeoffset
	77520       	# Cylinders according to firmware.
	16          	# Heads according to firmware.
	63          	# Sectors according to firmware.
	5PV09ZED    	# Disk ident.

Seek times:
	Full stroke:	  250 iter in   8.014669 sec =   32.059 msec
	Half stroke:	  250 iter in   6.244615 sec =   24.978 msec
	Quarter stroke:	  500 iter in   9.976294 sec =   19.953 msec
	Short forward:	  400 iter in   3.585714 sec =    8.964 msec
	Short backward:	  400 iter in   2.533780 sec =    6.334 msec
	Seq outer:	 2048 iter in   0.433188 sec =    0.212 msec
	Seq inner:	 2048 iter in   0.384784 sec =    0.188 msec
Transfer rates:
	outside:       102400 kbytes in  22.705212 sec =     4510 kbytes/sec
	middle:        102400 kbytes in  22.845457 sec =     4482 kbytes/sec
	inside:        102400 kbytes in  22.782483 sec =     4495 kbytes/sec

Steve

stephenw10

But what happens if the disk fails? Will the box still boot? If not then you'd be better off just booting from the HD. At least then you can update the firmwhere without having to modify the scripts again.

Steve

Steve Evans

Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure. If the disk fails, just pull it from the rear of the box, then pfSense can be up and running while you wait for a new disk to arrive off eBay, plug in the new one and carry on, albeit having lost some logs and the (not very valuable) cached data which is probably stale by that point anyway.

If fact, even if somebody tells me that an SSD is fast by comparison, I'll probably still stick with the CF for the OS, and the SSD for /var. Now that I have this working, I'm considering a 32G SSD drive as they're cheap, and spread over that size, the amount of writing my pfSense installation will do will probably see it last well.

Steve

stephenw10

@Steve:

Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure.

Ok, you've clearly thought this through.
Personally the ability to upgrade the firmware without having to re-edit the scripts outweighs the advantages offered by running from flash. That's just my situation though.

Steve

Steve Evans

Hi Steve,

What scripts are you referring to?

I run bacula on my home network, and my pfsense box has been added to that regime, so I could perform an upgrade and then restore modified scripts if desired.

Steve

stephenw10

@Steve:

What scripts are you referring to?

/etc/rc.embedded

If you can restore it easily enough then I guess it's less of an issue. Having more options is always good.

Steve

Steve Evans

Thanks to a post HERE by stephenw10 I've realised that my hard disk didn't have DMA enabled.

[2.1-RELEASE][admin@pfsense]/root(18): atacontrol mode ad1
current mode = PIO4 
[2.1-RELEASE][admin@pfsense]/root(19): atacontrol mode ad1 UDMA6
current mode = UDMA33 
[2.1-RELEASE][admin@pfsense]/root(20): diskinfo -tv /dev/ad1
/dev/ad1
	512         	# sectorsize
	40007761920 	# mediasize in bytes (37G)
	78140160    	# mediasize in sectors
	0           	# stripesize
	0           	# stripeoffset
	77520       	# Cylinders according to firmware.
	16          	# Heads according to firmware.
	63          	# Sectors according to firmware.
	5PV09ZED    	# Disk ident.

Seek times:
	Full stroke:	  250 iter in   7.989798 sec =   31.959 msec
	Half stroke:	  250 iter in   6.254760 sec =   25.019 msec
	Quarter stroke:	  500 iter in   9.845523 sec =   19.691 msec
	Short forward:	  400 iter in   3.577064 sec =    8.943 msec
	Short backward:	  400 iter in   2.471400 sec =    6.179 msec
	Seq outer:	 2048 iter in   0.187117 sec =    0.091 msec
	Seq inner:	 2048 iter in   0.332214 sec =    0.162 msec
Transfer rates:
	outside:       102400 kbytes in   3.476458 sec =    29455 kbytes/sec
	middle:        102400 kbytes in   3.863341 sec =    26506 kbytes/sec
	inside:        102400 kbytes in   5.491517 sec =    18647 kbytes/sec

Up to six times the speed. I'm happy with that!

Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.

Steve

demco

@Steve:

Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.

Steve

Setting these sysctl variables, in loader.conf.local, didn't enable DMA?

hw.ata.wc="1"
hw.ata.atapi_dma="1"
hw.ata.ata_dma="1"

stephenw10

Doing that enables DMA globally which includes the CF card. Since the CF-IDE adapter in the firebox doesn't support DMA it will fail to boot with pages of DMA errors.

Looks like the Shellcmd package is the way to go, no way of selectively enabling DMA per device.
Have you tried any BIOS DMA settings?

Steve

Steve Evans

Hi Steve,

Indeed, the issue here was that I wanted to selectively enable DMA on the hard disk only.

No, I've not looked at the BIOS. With a current uptime of 50 days I'm loathed to reboot it to be honest. What BIOS settings are there?

Steve

stephenw10

There are options for dma mode for primary master and slave separately. However those options were originally hidden and they are both set to UDMA disabled by default. Since udma still seems to be available maybe those options are not implemented. Easy test though.

Steve

sg1

Now I followed the post (which is exactly what I wanted to do as well) but for whatever reason, I have two /var still mounted. I have rebooted it a couple of times to see if I can see where it occurs or why, but have not been able to figure it out. Any suggestions I can check as to why? Otherwise, it works perfectly. df -h is below:
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/pfsense0 442M 379M 28M 93% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ufs/cf 49M 1.4M 44M 3% /cf
/dev/ad1s1a 36G 10k 33G 0% /var
/dev/md0 38M 90k 35M 0% /tmp
/dev/md1 57M 17M 35M 33% /var
devfs 1.0k 1.0k 0B 100% /var/dhcpd/dev

Thanks for any help!

stephenw10

What does your modified rc.embedded look like?
You should see the various messages from the modified script in the boot log, are you seeing those? Are they correct?

Steve

Steve Evans

The memory device /dev/md1 is still being created/mounted so as Stephen says you've got an issue with the startup script.

Steve

sg1

No. I see now, that it is incorrect. I still have the original being mounted due to it. My question is, where specifically in the original rc.embedded do I change with the Steve's changes as there are other calls in the script that I am not sure are needed or not. Apologies, just new to the pfsense and Watchguard platforms so I am sure my questions are on the the dumber side…. The original rc.embedded is below:

rc.embedded - embedded system specific startup information

For pfSense

Size of /tmp

USE_MFS_TMP_SIZE=/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'

if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then
tmpsize="${USE_MFS_TMP_SIZE}m"
else
tmpsize="40m"
fi

Size of /var

USE_MFS_VAR_SIZE=/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/ cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'
if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then
varsize="${USE_MFS_VAR_SIZE}m"
else
varsize="60m"
fi

Run some initialization routines

[ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start

echo -n "Setting up memory disks…"
mdmfs -S -M -s ${tmpsize} md /tmp
mdmfs -S -M -s ${varsize} md /var

Create some needed directories

/bin/mkdir -p /var/db

Ensure vi's recover directory is present

/bin/mkdir -p /var/tmp/vi.recover/
echo " done."

Steve Evans

Here is the complete patched /etc/rc.embedded.

#!/bin/sh
#
# rc.embedded - embedded system specific startup information
# For pfSense

# Size of /tmp
USE_MFS_TMP_SIZE=`/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'`
if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then
	tmpsize="${USE_MFS_TMP_SIZE}m"
else
	tmpsize="40m"
fi

# Size of /var
USE_MFS_VAR_SIZE=`/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'`
if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then
	varsize="${USE_MFS_VAR_SIZE}m"
else
	varsize="60m"
fi

# Run some initialization routines
[ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start

echo -n "Setting up memory disks..."
mdmfs -S -M -s ${tmpsize} md /tmp

# If a hard disk is installed then mount that on /var
# otherwise use a ramdisk
harddisk="/dev/ad1s1a"
if [ -c $harddisk ]
then
	echo -n "Using /var physical disk..."
	mount -o noatime $harddisk /var

	# Ensure /var/run is removed on boot before daemons are started
	# It should arguably be a tmpfs, but this works fine
	rm -r /var/run

	# sshd won't start if /var/empty exists on boot
	rm -r /var/empty
else
	echo -n "Using /var memory disk..."
	mdmfs -S -M -s ${varsize} md /var

	# Create some needed directories
	/bin/mkdir -p /var/db

	# Ensure vi's recover directory is present
	/bin/mkdir -p /var/tmp/vi.recover/
fi

echo " done."

sg1

Thank you so much Steve! Works perfectly. Thanks for your patience….

atrocity

only my result of a x5500e with a ide SSD from CWC :

/dev/ad1s1a
512 # sectorsize
57872344064 # mediasize in bytes (53G)
113031922 # mediasize in sectors
0 # stripesize
40448 # stripeoffset
112134 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
OW140822AS1514144 # Disk ident.

Seek times:
Full stroke: 250 iter in 0.040570 sec = 0.162 msec
Half stroke: 250 iter in 0.037379 sec = 0.150 msec
Quarter stroke: 500 iter in 0.125276 sec = 0.251 msec
Short forward: 400 iter in 0.093555 sec = 0.234 msec
Short backward: 400 iter in 0.096201 sec = 0.241 msec
Seq outer: 2048 iter in 0.169674 sec = 0.083 msec
Seq inner: 2048 iter in 0.123082 sec = 0.060 msec
Transfer rates:
outside: 102400 kbytes in 1.124830 sec = 91036 kbytes/sec
middle: 102400 kbytes in 1.128009 sec = 90779 kbytes/sec
inside: 102400 kbytes in 1.125042 sec = 91019 kbytes/sec

seems that i don't have to tune anything.
I will make the same test with a x750e soon, will let you know if somebody is interrested :)
Regards

stephenw10

What tool are you testing that with? Did you enable TRIM? Be interesting to see how those figures vary over time once you've moved a few gigs across the drive.

Steve

atrocity

hi,

i simply use this :

diskinfo -tv /dev/ad1

Trim … i did a try to enable it, but our server don't want it.
Don't remember the problem or the error, sorry.