UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE

JonathanLee

Swap getting some use. So now you know it works, do not use it nonstop, or plan to abuse it, you will kill drives, this is the emergency tank, and you have to understand why you need this before just adding it and using it like RAM. I use it for one thing updates so snort and clamav don't crash every once and a while they update at the same time.

JonathanLee

Part2:

GET DUMP TO STILL WORK:

if you notice there is a SWAP already on the freebsd drive
is it ada0s3b

WARNING DO NOT MESS UP OR YOU CAN BONK UP STUFF BAD HERE AND THAT WOULD REQUIRE A FULL REINSTALL.

Geom name: ada0s3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 249406126
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s3a
   Mediasize: 120590425600 (112G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339747328
   Mode: r1w1e1
   rawtype: 27
   length: 120590425600
   offset: 8192
   type: freebsd-zfs
   index: 1
   end: 235528190
   start: 16
2. Name: ada0s3b
   Mediasize: 7105150976 (6.6G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 120930172928
   Mode: r0w0e0
   rawtype: 1
   length: 7105150976
   offset: 120590433792
   type: freebsd-swap
   index: 2
   end: 249405438
   start: 235528191
Consumers:
1. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2

So I want to use ada0s3b for just dump now and still have the swap on the usb.

Here is how.

Create a cron job
@reboot dumpon /dev/ada0s3b
copy the /etc/rc.dumpon to /etc/rc.dumpon.old
edit the /etc/rc.dumpon to just have

#!/bin/sh
#
# rc.dumpon
#
# part of pfSense (https://www.pfsense.org)
# Copyright (c) 2004-2013 BSD Perimeter
# Copyright (c) 2013-2016 Electric Sheep Fencing
# Copyright (c) 2014-2023 Rubicon Communications, LLC (Netgate)
# All rights reserved.
#
# Based on src/etc/rc.d/dumpon from FreeBSD
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ln -fs /dev/ada0s3b /dev/dumpdev
echo "OK ADAPTED ADA0S3B ONLINE"
return 0

Save and reboot it should now use swap on the usb drive and crash reports will still show up.

If needed here is the old file it needs to return the dump location and a value so that savecore knows where to save the core dumps, you see here the code is all depended on the fstab file that we adapted with .eli so it wont work with that anymore so we have to adapt it.
OLD CODE:

#!/bin/sh
#
# rc.dumpon
#
# part of pfSense (https://www.pfsense.org)
# Copyright (c) 2004-2013 BSD Perimeter
# Copyright (c) 2013-2016 Electric Sheep Fencing
# Copyright (c) 2014-2023 Rubicon Communications, LLC (Netgate)
# All rights reserved.
#
# Based on src/etc/rc.d/dumpon from FreeBSD
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

dumpon_try()
{
	if /sbin/dumpon "${1}" ; then
		# Make a symlink in devfs for savecore
		echo "Using ${1} for dump device."
		ln -fs "${1}" /dev/dumpdev
		return 0
	fi
	echo "Unable to specify $1 as a dump device."
	return 1
}

# Enable dumpdev so that savecore can see it. Enable it
# early so a crash early in the boot process can be caught.
#
while read dev mp type more ; do
	[ "${type}" = "swap" ] || continue
	[ -c "${dev}" ] || continue
	dumpon_try "${dev}" && works=true
done </etc/fstab
if [ "${works}" != "true" ]; then
	# fstab entries did not work, try swap labels
	for dev in /dev/label/swap*; do
		if [ ! -e "${dev}" ]; then
			continue;
		fi
		dumpon_try "${dev}" && works=true
	done
	if [ "${works}" != "true" ]; then
		echo "No suitable dump device was found." 1>&2
		exit
	fi
fi

Now it works with both core dumps and swap.
Happy Holidays

jimp

When you run gpart add, use -l swap<something> to give the partition a name and then only reference it by that name. Never reference removable disks/slices/partitions by their device names since those can change depending on the order they are probed. Really any disk if you can help it, but especially not removable ones.

Just don't use a low number since those might conflict with swap already on existing disks. But if you name it something like swap99 or swapUSB that should still work.

By using that label name you shouldn't have to modify rc.dumpon either since it checks any disk label starting with swap* if it can't find a suitable dump partition in fstab.

If your old swap partition still had a device entry in fstab then the install must not have been recent. You could add a label with gpart or glabel. The installer names them starting with swap0 now, so a mirror might have swap0 and swap1 for example.

JonathanLee

@jimp said in UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE:

glabel

Thanks

"glabel" I will fix this now

JonathanLee

@jimp

Ok so the swapUSB lable is added how do I add this to the fstab file now so it can be mounted? /dev shows no label folder. Goal is to only use swapUSB for swap and the onboard ada0s3b for dump.

I do not know how to add a label to ada0s3b as this is the pfsense drive I do not really want to touch it. Is there a vetted command I can do to add the label that is needed so it can be used?

the USB uses .eli also so would I need to do swapUSB.eli?

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 250069679
first: 1
entries: 4
scheme: MBR
Providers:
1. Name: ada0s1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 512
   Mode: r0w0e0
   efimedia: HD(1,MBR,00000000,0x1,0x82000)
   rawtype: 239
   length: 272629760
   offset: 512
   type: efi
   index: 1
   end: 532480
   start: 1
2. Name: ada0s2
   Mediasize: 67108864 (64M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272630272
   Mode: r0w0e0
   efimedia: HD(2,MBR,00000000,0x82001,0x20000)
   rawtype: 11
   length: 67108864
   offset: 272630272
   type: fat32
   index: 2
   end: 663552
   start: 532481
3. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2
   efimedia: HD(3,MBR,00000000,0xa2001,0xedda2af)
   attrib: active
   rawtype: 165
   length: 127695937024
   offset: 339739136
   type: freebsd
   index: 3
   end: 250069679
   start: 663553
Consumers:
1. Name: ada0
   Mediasize: 128035676160 (119G)
   Sectorsize: 512
   Mode: r1w1e3

Geom name: ada0s3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 249406126
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s3a
   Mediasize: 120590425600 (112G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339747328
   Mode: r1w1e1
   rawtype: 27
   length: 120590425600
   offset: 8192
   type: freebsd-zfs
   index: 1
   end: 235528190
   start: 16
2. Name: ada0s3b
   Mediasize: 7105150976 (6.6G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 120930172928
   Mode: r0w0e0
   rawtype: 1
   length: 7105150976
   offset: 120590433792
   type: freebsd-swap
   index: 2
   end: 249405438
   start: 235528191
Consumers:
1. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 250069639
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,25749237-bf08-11ef-9a7e-90ec770dda25,0x800,0x1000000)
   rawuuid: 25749237-bf08-11ef-9a7e-90ec770dda25
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swapUSB
   length: 8589934592
   offset: 1048576
   type: freebsd-swap
   index: 1
   end: 16779263
   start: 2048
Consumers:
1. Name: da0
   Mediasize: 128035676160 (119G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2

jimp

The labels are in /dev/gpt/, so replace the /dev/<device> path in fstab with /dev/gpt/swapUSB

JonathanLee

@jimp said in UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE:

/dev/gpt/swapUSB

sorry to bug you just to confirm

I ran this
gpart add -t freebsd-swap -a 1M -s 8GB -l swapUSB da0

now the label is no longer null

this has the entry
/dev/da0p1.eli none swap sw 0 0

change this to
/dev/gpt/swapUSB.eli

for /dev/ada0s3b that I have the link created to /dev/dumpdev

this as far as I know has no label on it so how would rc.dumpon find this dump can not use .eli so I have dump using the onboard SSD and not the usb. Should I make the label just USBswap that way it avoids it and uses the other entry?

I do not really want to use destroy to remove the label on /dev/ada0s3b

however this should have label swap0 for it so rc.dumpon can just locate that first?

JonathanLee

@jimp I have no dev/gpt/ showing in my filesystem because that is the usb right?

jimp

You shouldn't have to destroy the existing swap on the disk to change the label.

# glabel label swap0 /dev/ada0s3b

And then when you reboot it should be fine.

The labels will disappear from /dev/gpt/ when the partitions are mounted/in-use but you should see the right thing happening in swapinfo and checking the dumpdev symlink.

You shouldn't need to manually do anything other than adding the label and adjusting fstab.

JonathanLee

@jimp

You the Man!!

JonathanLee

@jimp

I have an issue after reboot

It shows both now with the swapinfo command

Shell Output - swapinfo -kh
Device              Size     Used    Avail Capacity
/dev/gpt/swapUSB.eli     8.0G       0B     8.0G     0%
/dev/label/swap0     6.6G       0B     6.6G     0%
Total                15G       0B      15G     0%

It should only show /dev/gpt/swapUSB.eli for external swap because that is the only entry in the fstab file. However, both are being listed. the swap0 I only want for dump use coredumps because it is onboard ssd with the pfsense software swap would kill the drive faster. This was my issue before I couldn’t do coredumps on swapUSB because dump can’t use .eli so I wanted to separate swap from the coredumps location. For external usb based swap I want to use .eli to protect it from memory leaks so no one walks off with the drive with everything on it. I do not want to overload the onboard stuff because it is an SSD with pfsense.

dump on is correct thank you this does not use .eli it works does core dumps to that location. Weird that swapinfo shows both now that the labels are added. When is this checked what else loads the info into swap ?

DiagnosticsCommand Prompt
Shell Output - dumpon -l
label/swap0

jimp

The boot script should only be using swap entries listed in /etc/fstab/ so make sure swap0 isn't in fstab at all. The rc.dumpon script doesn't need an entry in fstab so long as the label exists.

JonathanLee

@jimp
I only have one entry in fstab what is weird after a reboot if I do swapinfo it shows both now.. I tested it did a crash and I do get coredumps now on the right drive and that is amazing !! If fstab lists one swap why is it now adding the swap0 to it and not just the swapUSB?

(Swapinfo showing 2 different swaps)

(What is listed in fstab)

Eli is loading also with the correct partition and encrypting on logs

JonathanLee

mer

@JonathanLee FreeBSD has this idea of "device withering".
If your swap partition is NOT mounted/in use, you will likely see a "/dev/da0p2" AND a "/dev/gpt/whateverlabelyougaveit".
Then the first exclusive reference (think mount) makes everything else disappear.
if you do "swapon /dev/da0p2" the /dev/gpt disappears.
If you do "swapon /dev/gpt/whatever" the /dev/da0p2 will likely disappear (not always since this is basically raw device)
Your swapinfo showing both /dev/gpt/swapUSB.eli and /dev/label/swap0 indicate to me they are on 2 different physical devices.
If they are not different physical devices, you have something very odd going on.

JonathanLee

Yes swapusb is a usb and the other is the SSD on the device itself. Two different devices, also .Eli is in use so dump can not use that device only swap can

JonathanLee

@mer I wonder if I do a cron job on it and swapon set it to be specific for usb if it would work, I opened a redmine as the current rc.dumpon does not include a if Eli and or if usb for situations like this. It was not closed right away so it could be fixed with a simple if else situation on the code.

JonathanLee

Fix is to set swapoff on the other drive with a cron job no code is adapted. I am going to update my unofficial guide

mer

@JonathanLee Keep in mind "swapoff" may not immediately free up swap space. I think items in swap need to wind up getting released (basically the VM system recognizes no longer needed) before the swap device is actually taken offline.

castleapp

The unofficial guide for running swap over a USB drive in pfSense explains how to configure a USB drive as swap space. While it can help with memory shortages, it risks reduced performance and potential wear on the USB drive due to frequent read/write cycles. Use cautiously and consider alternatives like increasing RAM.