Using a hard-disk in a Watchguard Firebox X750e for cache/log storage

[Steve Evans]

If you install squid you probably want to turn on disk caching and also keep the logs, but using nanobsd really doesn't play well with that. I've therefore fitted a hard drive to my X750e (the details of which are covered in other posts, but which only cost me <£15) to hold the /var partition.

Be sure to fit the "slave" link to the hard drive before you mount it in the caddy or it won't work.

Once you've booted pfSense, get a shell, and execute the following commands.

Format the disk thus:

fdisk -I /dev/ad1
bsdlabel -w ad1s1
newfs -U /dev/ad1s1a

Mount the disk and create some needed directories.

mount /dev/ad1s1a /mnt
cd /mnt
/bin/mkdir -p db
/bin/mkdir -p tmp/vi.recover/

Enable writes to your CF card thus:

/etc/rc.conf_mount_rw

Then modify /etc/rc.embedded to conditionally replace the mounting of the /var ramdisk thus:

echo -n "Setting up memory disks..."
mdmfs -S -M -s ${tmpsize} md /tmp

# If a hard disk is installed then mount that on /var
# otherwise use a ramdisk
harddisk="/dev/ad1s1a"
if [ -c $harddisk ]
then
        echo -n "Using /var physical disk..."
        mount $harddisk /var

        # Ensure /var/run is removed on boot before daemons are started
        # It should arguably be a tmpfs, but this works fine
        rm -r /var/run

        # sshd won't start if /var/empty exists on boot
        rm -r /var/empty
else
        echo -n "Using /var memory disk..."
        mdmfs -S -M -s ${varsize} md /var

        # Create some needed directories
        /bin/mkdir -p /var/db

        # Ensure vi's recover directory is present
        /bin/mkdir -p /var/tmp/vi.recover/
fi

echo " done."

Then reboot and you should see that you now have a rather large /var partition.

df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/pfsense1    1.8G    1.0G    641M    62%    /
devfs                1.0k    1.0k      0B   100%    /dev
/dev/ufs/cf           49M    1.9M     43M     4%    /cf
/dev/md0              38M    598k     34M     2%    /tmp
/dev/ad1s1a           36G     21M     33G     0%    /var
devfs                1.0k    1.0k      0B   100%    /var/dhcpd/dev
procfs               4.0k    4.0k      0B   100%    /proc

You can then enable disk caching, logging etc.

Some packages, such as sarg will still try to write to the CF card, but rather that try to find all the references to locations under /usr/local I simply created a symbolic link to a new directory on /var:

cd /usr/local/
ln -s /var/sarg-reports sarg-reports

And then write protect the CF card thus:

/etc/rc.conf_mount_ro

Steve

[ghostshell]

I just installed a 2.5 in one of my x750e's and an SSD IDE in the other in order to not have to mess with CF cards anymore

[stephenw10]

Nice write-up, however I think I'm with ghostshell here. What is the advantage of doing this over running a full install booting from the hd?

Steve

[Steve Evans]

CF is more reliable for read-only access and a rotating-rust hard disk will last much longer than an SSD where a lot of writes are being performed. This combo therefore makes (cheap) sense for a proxy cache.

One downside is speed… I'm only getting about 4.5Mbytes/s IO on the hard-disk which is no faster than my broadband connection.

Has anybody tested SSD speed using diskinfo?

The following is from a 40G 5400rpm drive, a Seagate ST9402115A:

diskinfo -tv /dev/ad1
/dev/ad1
	512         	# sectorsize
	40007761920 	# mediasize in bytes (37G)
	78140160    	# mediasize in sectors
	0           	# stripesize
	0           	# stripeoffset
	77520       	# Cylinders according to firmware.
	16          	# Heads according to firmware.
	63          	# Sectors according to firmware.
	5PV09ZED    	# Disk ident.

Seek times:
	Full stroke:	  250 iter in   8.014669 sec =   32.059 msec
	Half stroke:	  250 iter in   6.244615 sec =   24.978 msec
	Quarter stroke:	  500 iter in   9.976294 sec =   19.953 msec
	Short forward:	  400 iter in   3.585714 sec =    8.964 msec
	Short backward:	  400 iter in   2.533780 sec =    6.334 msec
	Seq outer:	 2048 iter in   0.433188 sec =    0.212 msec
	Seq inner:	 2048 iter in   0.384784 sec =    0.188 msec
Transfer rates:
	outside:       102400 kbytes in  22.705212 sec =     4510 kbytes/sec
	middle:        102400 kbytes in  22.845457 sec =     4482 kbytes/sec
	inside:        102400 kbytes in  22.782483 sec =     4495 kbytes/sec

Steve

[stephenw10]

But what happens if the disk fails? Will the box still boot? If not then you'd be better off just booting from the HD. At least then you can update the firmwhere without having to modify the scripts again.

Steve

[Steve Evans]

Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure. If the disk fails, just pull it from the rear of the box, then pfSense can be up and running while you wait for a new disk to arrive off eBay, plug in the new one and carry on, albeit having lost some logs and the (not very valuable) cached data which is probably stale by that point anyway.

If fact, even if somebody tells me that an SSD is fast by comparison, I'll probably still stick with the CF for the OS, and the SSD for /var. Now that I have this working, I'm considering a 32G SSD drive as they're cheap, and spread over that size, the amount of writing my pfSense installation will do will probably see it last well.

Steve

[stephenw10]

@Steve:

Yes, the rc.embedded change will only use the disk if the required partition is present. If not it will revert to the original ramdisk. So, this gives all the benefits of the solid state CF for long term reliability, but also provides for graceful failure.

Ok, you've clearly thought this through.
Personally the ability to upgrade the firmware without having to re-edit the scripts outweighs the advantages offered by running from flash. That's just my situation though.

Steve

[Steve Evans]

Hi Steve,

What scripts are you referring to?

I run bacula on my home network, and my pfsense box has been added to that regime, so I could perform an upgrade and then restore modified scripts if desired.

Steve

[stephenw10]

@Steve:

What scripts are you referring to?

/etc/rc.embedded

If you can restore it easily enough then I guess it's less of an issue. Having more options is always good.

Steve

[Steve Evans]

Thanks to a post HERE by stephenw10 I've realised that my hard disk didn't have DMA enabled.

[2.1-RELEASE][admin@pfsense]/root(18): atacontrol mode ad1
current mode = PIO4 
[2.1-RELEASE][admin@pfsense]/root(19): atacontrol mode ad1 UDMA6
current mode = UDMA33 
[2.1-RELEASE][admin@pfsense]/root(20): diskinfo -tv /dev/ad1
/dev/ad1
	512         	# sectorsize
	40007761920 	# mediasize in bytes (37G)
	78140160    	# mediasize in sectors
	0           	# stripesize
	0           	# stripeoffset
	77520       	# Cylinders according to firmware.
	16          	# Heads according to firmware.
	63          	# Sectors according to firmware.
	5PV09ZED    	# Disk ident.

Seek times:
	Full stroke:	  250 iter in   7.989798 sec =   31.959 msec
	Half stroke:	  250 iter in   6.254760 sec =   25.019 msec
	Quarter stroke:	  500 iter in   9.845523 sec =   19.691 msec
	Short forward:	  400 iter in   3.577064 sec =    8.943 msec
	Short backward:	  400 iter in   2.471400 sec =    6.179 msec
	Seq outer:	 2048 iter in   0.187117 sec =    0.091 msec
	Seq inner:	 2048 iter in   0.332214 sec =    0.162 msec
Transfer rates:
	outside:       102400 kbytes in   3.476458 sec =    29455 kbytes/sec
	middle:        102400 kbytes in   3.863341 sec =    26506 kbytes/sec
	inside:        102400 kbytes in   5.491517 sec =    18647 kbytes/sec

Up to six times the speed. I'm happy with that!

Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.

Steve

[demco]

@Steve:

Now to figure out the neatest way to invoke that command on boot. Adding it to startup using shellcmd is the simplest way, but I daresay there's a more appropriate "freebsd" place for it.

Steve

Setting these sysctl variables, in loader.conf.local, didn't enable DMA?

hw.ata.wc="1"
hw.ata.atapi_dma="1"
hw.ata.ata_dma="1"

[stephenw10]

Doing that enables DMA globally which includes the CF card. Since the CF-IDE adapter in the firebox doesn't support DMA it will fail to boot with pages of DMA errors.

Looks like the Shellcmd package is the way to go, no way of selectively enabling DMA per device.
Have you tried any BIOS DMA settings?

Steve

[Steve Evans]

Hi Steve,

Indeed, the issue here was that I wanted to selectively enable DMA on the hard disk only.

No, I've not looked at the BIOS. With a current uptime of 50 days I'm loathed to reboot it to be honest. What BIOS settings are there?

Steve

[stephenw10]

There are options for dma mode for primary master and slave separately. However those options were originally hidden and they are both set to UDMA disabled by default. Since udma still seems to be available maybe those options are not implemented. Easy test though.

Steve

[sg1]

Now I followed the post (which is exactly what I wanted to do as well) but for whatever reason, I have two /var still mounted.  I have rebooted it a couple of times to see if I can see where it occurs or why, but have not been able to figure it out.  Any suggestions I can check as to why?  Otherwise, it works perfectly.  df -h is below:
Filesystem          Size    Used  Avail Capacity  Mounted on
/dev/ufs/pfsense0    442M    379M    28M    93%    /
devfs                1.0k    1.0k      0B  100%    /dev
/dev/ufs/cf          49M    1.4M    44M    3%    /cf
/dev/ad1s1a          36G    10k    33G    0%    /var
/dev/md0              38M    90k    35M    0%    /tmp
/dev/md1              57M    17M    35M    33%    /var
devfs                1.0k    1.0k      0B  100%    /var/dhcpd/dev

Thanks for any help!

[stephenw10]

What does your modified rc.embedded look like?
You should see the various messages from the modified script in the boot log, are you seeing those? Are they correct?

Steve

[Steve Evans]

The memory device /dev/md1 is still being created/mounted so as Stephen says you've got an issue with the startup script.

Steve

[sg1]

No.  I see now, that it is incorrect.  I still have the original being mounted due to it.  My question is, where specifically in the original rc.embedded do I change with the Steve's changes as there are other calls in the script that I am not sure are needed or not.  Apologies, just new to the pfsense and Watchguard platforms so I am sure my questions are on the the dumber side….  The original rc.embedded is below:

rc.embedded - embedded system specific startup information

For pfSense

Size of /tmp

USE_MFS_TMP_SIZE=/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'

if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then
        tmpsize="${USE_MFS_TMP_SIZE}m"
else
        tmpsize="40m"
fi

Size of /var

USE_MFS_VAR_SIZE=/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/                                                                                 cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'
if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then
        varsize="${USE_MFS_VAR_SIZE}m"
else
        varsize="60m"
fi

Run some initialization routines

[ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start

echo -n "Setting up memory disks…"
mdmfs -S -M -s ${tmpsize} md /tmp
mdmfs -S -M -s ${varsize} md /var

Create some needed directories

/bin/mkdir -p /var/db

Ensure vi's recover directory is present

/bin/mkdir -p /var/tmp/vi.recover/
echo " done."

[Steve Evans]

Here is the complete patched /etc/rc.embedded.

#!/bin/sh
#
# rc.embedded - embedded system specific startup information
# For pfSense

# Size of /tmp
USE_MFS_TMP_SIZE=`/usr/bin/grep use_mfs_tmp_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'`
if [ ! -z ${USE_MFS_TMP_SIZE} ] && [ ${USE_MFS_TMP_SIZE} -gt 0 ]; then
	tmpsize="${USE_MFS_TMP_SIZE}m"
else
	tmpsize="40m"
fi

# Size of /var
USE_MFS_VAR_SIZE=`/usr/bin/grep use_mfs_var_size /cf/conf/config.xml | /usr/bin/cut -f2 -d'>' | /usr/bin/cut -f1 -d'<'`
if [ ! -z ${USE_MFS_VAR_SIZE} ] && [ ${USE_MFS_VAR_SIZE} -gt 0 ]; then
	varsize="${USE_MFS_VAR_SIZE}m"
else
	varsize="60m"
fi

# Run some initialization routines
[ -f /etc/rc.d/uzip ] && /etc/rc.d/uzip start

echo -n "Setting up memory disks..."
mdmfs -S -M -s ${tmpsize} md /tmp

# If a hard disk is installed then mount that on /var
# otherwise use a ramdisk
harddisk="/dev/ad1s1a"
if [ -c $harddisk ]
then
	echo -n "Using /var physical disk..."
	mount -o noatime $harddisk /var

	# Ensure /var/run is removed on boot before daemons are started
	# It should arguably be a tmpfs, but this works fine
	rm -r /var/run

	# sshd won't start if /var/empty exists on boot
	rm -r /var/empty
else
	echo -n "Using /var memory disk..."
	mdmfs -S -M -s ${varsize} md /var

	# Create some needed directories
	/bin/mkdir -p /var/db

	# Ensure vi's recover directory is present
	/bin/mkdir -p /var/tmp/vi.recover/
fi

echo " done."

[sg1]

Thank you so much Steve!  Works perfectly.  Thanks for your patience….