sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE

Sergei_Shablovsky

@stephenw10 said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

The ISO image doesn't boot memstick, it's not a hybrid image.

Hm. Not agree.
If I remember my experience correctly, I successfully create bootable USB-drive with pfSense CE from official.iso image just by Rufus (with no any extra settings in Rufus at all).
But with all respects, I’ll try again one time (may be Rufus make some smart converting…).

[Update]
I triple check, - on:

pfSense CE/Plus;
FreeBSD 14-REL/CURR
FreeBSD 15-CURR

Rufus v 4.4.2103 SUCCESSFULLY CREATE BOOTABLE (for both old BIOS-only nowadays UEFI systems) USB-pendrive:it make this AUTOMATICALLY, only show the warning message that ~”ISO/File mode not in option because image You provide not support it, so DD method of creating would be used instead”.

So, You are wrong in decision.

The new Net Installer will always offer the current image:
https://www.netgate.com/blog/public-beta-of-new-installer-for-pfsense-software

Thank You!

I read this news. But on March 2024 this is ONLY for Netgate appliances.

Moreover, I prefer the way when I have “rescue” USB-drive that contains all that need for “first-aid tools”: LATEST IMAGE, LATEST GOOD CONFIG, LICENCE (if needed) and able to making UNATTENDED INSTALL just booting from it.

This mean if something happened with server drive itself (which makes previous .conf file unavailable) or FreeBSD system itself, I just make ONE-TIME START FROM USB in BIOS, pfSense installer make UNATTENDED install, copy .conf from USB-drive to server’s drive in /conf dir, and reboot the server.
After that all next times server booting normally from internal HDD/SDD drives.

May be this “unattended rescue mode” would be realized in future versions of pfSense install. Because 8Gb flash drive cost a few $ and no problem to install it inside Netgate appliance on a factory. I hope ;)
Of coarse, rollback to “previous successfully start” by using ZFS snapshot are also good, but only up to point where internal SDD/HDD fail of ZFS structure seriously damaged.
This like “low” and “hi” copy of BIOS: you are able to update to latest release by special manual procedure + you are able to quickly restore working copy to original state.

Because for now this new installer just seek for Netgate appliance NDI and then proceed to CE or Plus installation.
May be next step would be (like in most other proprietary systems) - seeking the license/subscription key file on the same media where new installer start from. In that particular case, end user need some sort (.txt, .licences, etc) of license file on the same bootable USB-drive to be able to make fresh pfSense install on bare metal server.

[UPDATE]
At general (especially if analyzing official docs) looks like pfSense DevTeam have some sort of “fragmented view” on a resolving most common issues.
Just one example: troubleshooting with pfSense system upgrade they’re suggesting to resolving on different levels (cleaning caches, determining accessibility of official Netgate’s repo sites, clean browser cache, check free space on a drive, UEFI device with non-UEFI system, versions of pkg-update utility, connect timeout, serts rehashing, keys mismatched, segmentation fault, etc, etc … WHY NOT MAKING GOOD UPDATE SH/BASH/ZSH SCRIPT !!!?????
If You able to describe this in official docs, You certainty able to write a script!
And this still for a years….!

Back to my question:

I just try to create automatic sh script to AUTOMATICALLY creating bootable USB-drive with LATEST pfSense CE installer version:

download image and create bootable drive;
copy current .config file from existed and working well pfSense system /conf to root of the USB-drive;

elvisimprsntr

I just download the memstick img, use BelenaEtcher, boot and install on SSD

Works automagically!

Really shouldn't be running pfSense (or any OS) off thumb drives.

Thumb drive flash memory does not have the same error correction, wear leveling, and PE cycles as a SSD.

Sergei_Shablovsky

@elvisimprsntr said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

I just download the memstick img, use BelenaEtcher, boot and install on SSD

Works automagically!

I just need sh script to avoid this “download the memstik img, using Balena Etcher” procedure, script must AUTOMATICALLY making bootable “rescue” USB-drive INSIDE my server on crone job or manually.
(Please read previous updated my answer to @stephenw10)

Really shouldn't be running pfSense (or any OS) off thumb drives.

Thumb drive flash memory does not have the same error correction, wear leveling, and PE cycles as a SSD.

Agree. Of course.

elvisimprsntr

@Sergei_Shablovsky

I just run a daily cron job to backup the config.xml file to my TrueNAS server. I know pfsense now has some sort of backup mechanism, but I would rather have it local since the file contains credentials.


#!/bin/sh
VERSION=`cat /etc/version`
DATE=`date +%Y%m%d`
FILE="config_`hostname -s`_"$DATE"_"$VERSION".xml"
NAS="nas-1"
PATH="/mnt/data/Software/pfsense"

# mkdir /media/usb

/sbin/mount_msdosfs /dev/da0s1 /media/usb
if [ "$?" -eq "0" ]; then
	echo "USB found"
	/bin/cp /cf/conf/config.xml /media/usb/$FILE
	echo "Backup $FILE created"
	/usr/bin/find /media/usb/ -name "config_*.xml" -mtime +365 -exec rm {} \;
	/sbin/umount /media/usb	
else
	echo "USB not found"
fi	

/sbin/ping -c 3 $NAS > /dev/null 2>&1
	if [ $? -eq 0 ]; then
		echo "$NAS found" 
		/usr/bin/scp /cf/conf/config.xml root@$NAS:$PATH/$FILE
		/usr/bin/scp /root/pkg_check.php root@$NAS:$PATH/
		/usr/bin/scp /root/att_cidr.sh root@$NAS:$PATH/
		/usr/bin/scp /root/backup.sh root@$NAS:$PATH/
		echo "Backup $FILE copied to $NAS"
	else
		echo "$NAS not found"
	fi
	
# install cron package and add cron job
# 0 4 * * Sun /bin/sh /root/backup.sh > /dev/null

stephenw10

I guess you would need to pull the file list from: https://atxfiles.netgate.com/mirror/downloads/
Then parse the latest version and compare it with something cached.

Sergei_Shablovsky

@stephenw10 said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

I guess you would need to pull the file list from: https://atxfiles.netgate.com/mirror/downloads/
Then parse the latest version and compare it with something cached.

Exactly!
Thank You so much!

Is this https://atxfiles.netgate.com/mirror/downloads/ path fixed constant ?

stephenw10

It has been that for a long time. I can't promise 100% it will remain that but I see no reason to change it.

The script could throw an error if it can't see any files.

Sergei_Shablovsky

@stephenw10 said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

It has been that for a long time. I can't promise 100% it will remain that but I see no reason to change it.

Nice news, thank You!

[UPDATE]
The same url (+extra for statistics collection) used on official pfSense CE download page.

The script could throw an error if it can't see any files.

I think that special package would be useful for this: package intended for all users who have ability to install 8Gb USB-memstick inside their pfSense server/device/appliance or in free USB port.

Like on this image:

In settings of this package would be:

this URL to Netgate download directory;
filename scheme;
version;
frequency (in a days/weeks/months) of creating this “rescue boot drive”;
check box to also initiate “Auto Config Backup” on a Netgate remote servers;
checkbox to also initiate “Backup / Restore” to /backup-restore on this “rescue drive”;
checkbox to also initiate “Backup Files/Dir” (if installed) to /backup-filedir on this “rescue drive”;

Reasonable?

Sergei_Shablovsky

@elvisimprsntr said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

@Sergei_Shablovsky

I just run a daily cron job to backup the config.xml file to my TrueNAS server. I know pfsense now has some sort of backup mechanism, but I would rather have it local since the file contains credentials.


#!/bin/sh
VERSION=`cat /etc/version`
DATE=`date +%Y%m%d`
FILE="config_`hostname -s`_"$DATE"_"$VERSION".xml"
NAS="nas-1"
PATH="/mnt/data/Software/pfsense"

# mkdir /media/usb

/sbin/mount_msdosfs /dev/da0s1 /media/usb
if [ "$?" -eq "0" ]; then
	echo "USB found"
	/bin/cp /cf/conf/config.xml /media/usb/$FILE
	echo "Backup $FILE created"
	/usr/bin/find /media/usb/ -name "config_*.xml" -mtime +365 -exec rm {} \;
	/sbin/umount /media/usb	
else
	echo "USB not found"
fi	

/sbin/ping -c 3 $NAS > /dev/null 2>&1
	if [ $? -eq 0 ]; then
		echo "$NAS found" 
		/usr/bin/scp /cf/conf/config.xml root@$NAS:$PATH/$FILE
		/usr/bin/scp /root/pkg_check.php root@$NAS:$PATH/
		/usr/bin/scp /root/att_cidr.sh root@$NAS:$PATH/
		/usr/bin/scp /root/backup.sh root@$NAS:$PATH/
		echo "Backup $FILE copied to $NAS"
	else
		echo "$NAS not found"
	fi
	
# install cron package and add cron job
# 0 4 * * Sun /bin/sh /root/backup.sh > /dev/null

Thank You so much for sharing script!

I use it as part of mine. ;)

Why you use pinging the remote host instead of checking if certain remote path exist (or checking the success of this remote path creating)?

elvisimprsntr

@Sergei_Shablovsky said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

Why you use pinging the remote host instead of checking if certain remote path exist (or checking the success of this remote path creating)?

I have two pfSense sites on a Tailscale MESH VPN, one behind double NAT.
I use the same script to backup the remote site to my local NAS.
Sometimes it take a few pings before the NAS is reachable via Tailscale.

Sergei_Shablovsky

@elvisimprsntr said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

@Sergei_Shablovsky said in sh script to create bootable USB-drive with LATEST OFFICIAL REL of pfSense CE:

Why you use pinging the remote host instead of checking if certain remote path exist (or checking the success of this remote path creating)?

I have two pfSense sites on a Tailscale MESH VPN, one behind double NAT.
I use the same script to backup the remote site to my local NAS.

I have decision that procedure of correct and flawless backup/restore pfSense configuration still are so called “headpain point” for most of all pfSense users

And even provided “rollback to last good configuration by using ZFS snapshots feature” - not so help with this: this ZFS-rollback” really good in the middle of working system, but not good if you need quick restore after hardware failure when needed to setup fresh on bare metal another server.

Several times I see how after disc crash in Netgate Appliance and replace disc on same Appliance, procedure of “complete restore from last good ACB configuration from remote Netgate servers” not flawless: sometime some packages not installed for unknown reasons, and hw rebooting between some packages still needed…

Sad bud true…

Sometimes it take a few pings before the NAS is reachable via Tailscale.

Why You not using FreeBSD famous net/rclone, backup/zapzend, backup/zfs_autobackup, backup/sanoid and syncoid?

Each of this solution give You more flexibility, because You not only need to backup one pfSense config.xml, but may be a bunch of other scripts and edited BSD system files with custom settings.