UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE

JonathanLee

Hello, Fellow Netgate community members,

I researched this for a while and worked with many helpful people to compile this short simple guide.

Background on SWAP: Swap is basically a hypothetical emergency gas take for RAM. Some Netgate systems have embedded ram and can run out of ram during ClamAV updates, or other memory hungry tasks, and this is why watchdog restarts items that crash because they ran out of memory, Snort IPS/IDS during updates takes up tons of memory. So, SSDs come into play here they are very fast, and affordable today was not always the case. So, if you are wondering can I just use a USB adapted for M.2 and use an external SSD drive for SWAP yes you can but there are risks and other issues.

Let's talk about the issues I have found doing this:

USB2.0 is slow versus an onboard M.2 port.
True and nothing is stopping you from using the onboard SSD with a swap partition except this you can kill the SSD drive as they are not designed for use with SWAP. SWAPPING has many rewrites and can cook an SSD. Again, if you do not care about swapping out your SSD when it fails and reinstalling pfSense by all means go this route.
Memory leaking. This is a risk that can be corrected with the use of .eli on your fstab file. One note .eli is for swap and not for crash dumps, so dumpon -l will go into null.
When using USB swap the dumpon -l shows dev/null
Again, for some reason you can't use dumpon with freebsd-swap over .eli and on an external drive in pfSense. Might be a way to use a different swap for dump. Research in progress here.
What if I gpart and destroy my main drive?
This is a possibility if you do not know what your running gpart on. I can show you steps to avoid this completely.

So with this in mind here is the guide.

Supplies a cache based SSD with DRAM onboard
a USB m.2 adapter

Changes needed.

First format the USB drive with gpart and give it a freebsd-swap partition filesystem.

WARNING fist run gpart show and look at the normalized information before you plug in your USB drive

My Netgate 2100-MAX normal drive values before adding the USB.

You do not want to mess with ada0 that is the onboard stuff and it is working hard to protect your systems, so do not mess with it.
Advanced users this is where you would find your ada0s3b swap partition.

plug the drive in and run
gpart show again you want to find something like da0 that is your usb drive.

Once you see da0 we need to configure it for use with swapping. I am using the 1M so the blocks are larger ref this for more information.
Ref:
https://docs.freebsd.org/en/books/handbook/disks/
The above document uses ada DO NOT RUN commands with ada that is your pfSense.

gpart create -s GPT da0
gpart add -t freebsd-swap -a 1M -l swapUSB da0
gpart show da0
newfs -U /dev/da0p1

Side Note: The SWAP should only be 2x the size of your RAM to use size specific partition creation use "-s XGB"
Example of size specific

gpart add -t freebsd-swap -a 1M -s 8GB -l swapUSB da0

Bingo now you should see this when you run gpart show again

(take note of da0 with new swap)

Now lets adapt your fstab file

/dev/da0p1.eli		none	swap	sw		0	0

with use of .eli it encrypts the swap to protect from memory leaks

(make sure you do not have an extra line check this by running your arrow key to the end it should stop at 0 and not go below it save this)

reboot your system once it comes up check to see if the swap is running with use of

Shell Output - swapinfo
Device          512-blocks     Used    Avail Capacity
/dev/da0p1.eli   250066944   414792 249652152     0%

Your swap is running. Check in the dashboard it should also list your swap now.

JonathanLee

Swap getting some use. So now you know it works, do not use it nonstop, or plan to abuse it, you will kill drives, this is the emergency tank, and you have to understand why you need this before just adding it and using it like RAM. I use it for one thing updates so snort and clamav don't crash every once and a while they update at the same time.

JonathanLee

Part2:

GET DUMP TO STILL WORK:

if you notice there is a SWAP already on the freebsd drive
is it ada0s3b

WARNING DO NOT MESS UP OR YOU CAN BONK UP STUFF BAD HERE AND THAT WOULD REQUIRE A FULL REINSTALL.

Geom name: ada0s3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 249406126
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s3a
   Mediasize: 120590425600 (112G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339747328
   Mode: r1w1e1
   rawtype: 27
   length: 120590425600
   offset: 8192
   type: freebsd-zfs
   index: 1
   end: 235528190
   start: 16
2. Name: ada0s3b
   Mediasize: 7105150976 (6.6G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 120930172928
   Mode: r0w0e0
   rawtype: 1
   length: 7105150976
   offset: 120590433792
   type: freebsd-swap
   index: 2
   end: 249405438
   start: 235528191
Consumers:
1. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2

So I want to use ada0s3b for just dump now and still have the swap on the usb.

Here is how.

Create a cron job
@reboot dumpon /dev/ada0s3b
copy the /etc/rc.dumpon to /etc/rc.dumpon.old
edit the /etc/rc.dumpon to just have

#!/bin/sh
#
# rc.dumpon
#
# part of pfSense (https://www.pfsense.org)
# Copyright (c) 2004-2013 BSD Perimeter
# Copyright (c) 2013-2016 Electric Sheep Fencing
# Copyright (c) 2014-2023 Rubicon Communications, LLC (Netgate)
# All rights reserved.
#
# Based on src/etc/rc.d/dumpon from FreeBSD
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ln -fs /dev/ada0s3b /dev/dumpdev
echo "OK ADAPTED ADA0S3B ONLINE"
return 0

Save and reboot it should now use swap on the usb drive and crash reports will still show up.

If needed here is the old file it needs to return the dump location and a value so that savecore knows where to save the core dumps, you see here the code is all depended on the fstab file that we adapted with .eli so it wont work with that anymore so we have to adapt it.
OLD CODE:

#!/bin/sh
#
# rc.dumpon
#
# part of pfSense (https://www.pfsense.org)
# Copyright (c) 2004-2013 BSD Perimeter
# Copyright (c) 2013-2016 Electric Sheep Fencing
# Copyright (c) 2014-2023 Rubicon Communications, LLC (Netgate)
# All rights reserved.
#
# Based on src/etc/rc.d/dumpon from FreeBSD
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

dumpon_try()
{
	if /sbin/dumpon "${1}" ; then
		# Make a symlink in devfs for savecore
		echo "Using ${1} for dump device."
		ln -fs "${1}" /dev/dumpdev
		return 0
	fi
	echo "Unable to specify $1 as a dump device."
	return 1
}

# Enable dumpdev so that savecore can see it. Enable it
# early so a crash early in the boot process can be caught.
#
while read dev mp type more ; do
	[ "${type}" = "swap" ] || continue
	[ -c "${dev}" ] || continue
	dumpon_try "${dev}" && works=true
done </etc/fstab
if [ "${works}" != "true" ]; then
	# fstab entries did not work, try swap labels
	for dev in /dev/label/swap*; do
		if [ ! -e "${dev}" ]; then
			continue;
		fi
		dumpon_try "${dev}" && works=true
	done
	if [ "${works}" != "true" ]; then
		echo "No suitable dump device was found." 1>&2
		exit
	fi
fi

Now it works with both core dumps and swap.
Happy Holidays

jimp

When you run gpart add, use -l swap<something> to give the partition a name and then only reference it by that name. Never reference removable disks/slices/partitions by their device names since those can change depending on the order they are probed. Really any disk if you can help it, but especially not removable ones.

Just don't use a low number since those might conflict with swap already on existing disks. But if you name it something like swap99 or swapUSB that should still work.

By using that label name you shouldn't have to modify rc.dumpon either since it checks any disk label starting with swap* if it can't find a suitable dump partition in fstab.

If your old swap partition still had a device entry in fstab then the install must not have been recent. You could add a label with gpart or glabel. The installer names them starting with swap0 now, so a mirror might have swap0 and swap1 for example.

JonathanLee

@jimp said in UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE:

glabel

Thanks

"glabel" I will fix this now

JonathanLee

@jimp

Ok so the swapUSB lable is added how do I add this to the fstab file now so it can be mounted? /dev shows no label folder. Goal is to only use swapUSB for swap and the onboard ada0s3b for dump.

I do not know how to add a label to ada0s3b as this is the pfsense drive I do not really want to touch it. Is there a vetted command I can do to add the label that is needed so it can be used?

the USB uses .eli also so would I need to do swapUSB.eli?

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 250069679
first: 1
entries: 4
scheme: MBR
Providers:
1. Name: ada0s1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 512
   Mode: r0w0e0
   efimedia: HD(1,MBR,00000000,0x1,0x82000)
   rawtype: 239
   length: 272629760
   offset: 512
   type: efi
   index: 1
   end: 532480
   start: 1
2. Name: ada0s2
   Mediasize: 67108864 (64M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272630272
   Mode: r0w0e0
   efimedia: HD(2,MBR,00000000,0x82001,0x20000)
   rawtype: 11
   length: 67108864
   offset: 272630272
   type: fat32
   index: 2
   end: 663552
   start: 532481
3. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2
   efimedia: HD(3,MBR,00000000,0xa2001,0xedda2af)
   attrib: active
   rawtype: 165
   length: 127695937024
   offset: 339739136
   type: freebsd
   index: 3
   end: 250069679
   start: 663553
Consumers:
1. Name: ada0
   Mediasize: 128035676160 (119G)
   Sectorsize: 512
   Mode: r1w1e3

Geom name: ada0s3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 249406126
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s3a
   Mediasize: 120590425600 (112G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339747328
   Mode: r1w1e1
   rawtype: 27
   length: 120590425600
   offset: 8192
   type: freebsd-zfs
   index: 1
   end: 235528190
   start: 16
2. Name: ada0s3b
   Mediasize: 7105150976 (6.6G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 120930172928
   Mode: r0w0e0
   rawtype: 1
   length: 7105150976
   offset: 120590433792
   type: freebsd-swap
   index: 2
   end: 249405438
   start: 235528191
Consumers:
1. Name: ada0s3
   Mediasize: 127695937024 (119G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 339739136
   Mode: r1w1e2

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 250069639
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,25749237-bf08-11ef-9a7e-90ec770dda25,0x800,0x1000000)
   rawuuid: 25749237-bf08-11ef-9a7e-90ec770dda25
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swapUSB
   length: 8589934592
   offset: 1048576
   type: freebsd-swap
   index: 1
   end: 16779263
   start: 2048
Consumers:
1. Name: da0
   Mediasize: 128035676160 (119G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2

jimp

The labels are in /dev/gpt/, so replace the /dev/<device> path in fstab with /dev/gpt/swapUSB

JonathanLee

@jimp said in UNOFFICIAL GUIDE: RUN SWAP OVER USB DRIVE IN PFSENSE:

/dev/gpt/swapUSB

sorry to bug you just to confirm

I ran this
gpart add -t freebsd-swap -a 1M -s 8GB -l swapUSB da0

now the label is no longer null

this has the entry
/dev/da0p1.eli none swap sw 0 0

change this to
/dev/gpt/swapUSB.eli

for /dev/ada0s3b that I have the link created to /dev/dumpdev

this as far as I know has no label on it so how would rc.dumpon find this dump can not use .eli so I have dump using the onboard SSD and not the usb. Should I make the label just USBswap that way it avoids it and uses the other entry?

I do not really want to use destroy to remove the label on /dev/ada0s3b

however this should have label swap0 for it so rc.dumpon can just locate that first?

JonathanLee

@jimp I have no dev/gpt/ showing in my filesystem because that is the usb right?

jimp

You shouldn't have to destroy the existing swap on the disk to change the label.

# glabel label swap0 /dev/ada0s3b

And then when you reboot it should be fine.

The labels will disappear from /dev/gpt/ when the partitions are mounted/in-use but you should see the right thing happening in swapinfo and checking the dumpdev symlink.

You shouldn't need to manually do anything other than adding the label and adjusting fstab.

JonathanLee

@jimp

You the Man!!

JonathanLee

@jimp

I have an issue after reboot

It shows both now with the swapinfo command

Shell Output - swapinfo -kh
Device              Size     Used    Avail Capacity
/dev/gpt/swapUSB.eli     8.0G       0B     8.0G     0%
/dev/label/swap0     6.6G       0B     6.6G     0%
Total                15G       0B      15G     0%

It should only show /dev/gpt/swapUSB.eli for external swap because that is the only entry in the fstab file. However, both are being listed. the swap0 I only want for dump use coredumps because it is onboard ssd with the pfsense software swap would kill the drive faster. This was my issue before I couldn’t do coredumps on swapUSB because dump can’t use .eli so I wanted to separate swap from the coredumps location. For external usb based swap I want to use .eli to protect it from memory leaks so no one walks off with the drive with everything on it. I do not want to overload the onboard stuff because it is an SSD with pfsense.

dump on is correct thank you this does not use .eli it works does core dumps to that location. Weird that swapinfo shows both now that the labels are added. When is this checked what else loads the info into swap ?

DiagnosticsCommand Prompt
Shell Output - dumpon -l
label/swap0

jimp

The boot script should only be using swap entries listed in /etc/fstab/ so make sure swap0 isn't in fstab at all. The rc.dumpon script doesn't need an entry in fstab so long as the label exists.

JonathanLee

@jimp
I only have one entry in fstab what is weird after a reboot if I do swapinfo it shows both now.. I tested it did a crash and I do get coredumps now on the right drive and that is amazing !! If fstab lists one swap why is it now adding the swap0 to it and not just the swapUSB?

(Swapinfo showing 2 different swaps)

(What is listed in fstab)

Eli is loading also with the correct partition and encrypting on logs

JonathanLee

mer

@JonathanLee FreeBSD has this idea of "device withering".
If your swap partition is NOT mounted/in use, you will likely see a "/dev/da0p2" AND a "/dev/gpt/whateverlabelyougaveit".
Then the first exclusive reference (think mount) makes everything else disappear.
if you do "swapon /dev/da0p2" the /dev/gpt disappears.
If you do "swapon /dev/gpt/whatever" the /dev/da0p2 will likely disappear (not always since this is basically raw device)
Your swapinfo showing both /dev/gpt/swapUSB.eli and /dev/label/swap0 indicate to me they are on 2 different physical devices.
If they are not different physical devices, you have something very odd going on.

JonathanLee

Yes swapusb is a usb and the other is the SSD on the device itself. Two different devices, also .Eli is in use so dump can not use that device only swap can

JonathanLee

@mer I wonder if I do a cron job on it and swapon set it to be specific for usb if it would work, I opened a redmine as the current rc.dumpon does not include a if Eli and or if usb for situations like this. It was not closed right away so it could be fixed with a simple if else situation on the code.

JonathanLee

Fix is to set swapoff on the other drive with a cron job no code is adapted. I am going to update my unofficial guide

mer

@JonathanLee Keep in mind "swapoff" may not immediately free up swap space. I think items in swap need to wind up getting released (basically the VM system recognizes no longer needed) before the swap device is actually taken offline.