Clarification on using USB stick during installation to restore backup.
-
Hi
So changed storage in my home pfSense, and I made config backup before I started, made install usb stick using img with rufus, this had a visible fat32 partition with the efi folder inside it.
I made a conf folder and placed the backup config in there named config.xml.
I proceeded to install pfSense with this usb stick as the installer, and on first bootup could see it didnt restore the config.
Documentation here.
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html
I then noticed further down it says you can also restore during any bootup, but using a config folder instead of conf, this one did actually work.
So is it possible its a typo for the during install and should be config for that as well? or did I hit another problem?
pfSense CE 2.7.2
-
@chrcoluk said in Clarification on using USB stick during installation to restore backup.:
I made a conf folder and placed the backup config in there named config.xml.
I proceeded to install pfSense with this usb stick as the installer, and on first bootup could see it didnt restore the config.It's not really needed to incorporate your 'local' config.xml file.
The USB stick created with Rufus should boot, of course, and should be used to install a copy of pfSense on the local 'hard' drive.
Then all you have to do is : assign a 'LAN' interface (and maybe a WAN - if it doesn't work, no problem). As soon as the GUI fires up on the LAN, use any browser on a device on LAN, upload the config, and have pfSense reboot. When it comes back, everything is set.Normally ;) you should backup a copy of the config.xml very often. Incorporating this file into the bootable USB drive will get skipped as it is tedious .... so it will never be up to date the day you need it **.
Just download the config.xml on a PC, an d do so on anther PC also, or save a copy on some NAS, or a cloud drive, and you'll be fine.So, in short : download a copy from https://www.pfsense.org/download/ - don't touch this file, 'rufus' it to a USB drive 'as is'. Done.
I'm using myself this tool that runs on 'some Microsoft device' every day and grabs a copy of config.xml.
All I have to do now, is checking ones in a while that the folder uses contains a recent (24 hours max) copy of the config.cml. The tool will auto even purge xx days older copies.** Remember : if you have up to date copies, you'll never need them.
-
@gertjan Hi, this wasnt a query about backing up the config.
I of course do make regular backups, the query was about restoring the config during installation, and how it is documented on the netgate website.
The purpose of this is to avoid having to setup LAN access to restore via the GUI, as doing so automatically is much more convenient, thankfully I was able to fallback to the USB boot time restore method.
-
chrcoluk
I recently had a similar question. The documentation isn't very clear if it should be in the FAT32 partition or the FAT partition. After creating the install USB for our 5100 and 6100 devices, in Windows, it came up with three removable drives on my PC. I added a /conf (not /config) folder and put config.xml into it on both the FAT and FAT32 drives. For me they were EFISYS (E:), formatted FAT and PFSENSE (G:), formatted FAT32. That worked to restore the config during the new install, but I'm still not sure which one it actually used and the documentation isn't clear on it.
Seems like it would be easy to figure out with some testing, but since that worked for me, I just do it that way.
-
I'm using this one for backup (linux shellscript)
https://www.provya.com/blog/pfsense-making-automatic-backups-with-a-script/
https://github.com/blogmotion/bm-backup-pfsense
Works nicely .. tested on 2.5.2I briefly looked at the
https://github.com/KoenZomers/pfSenseBackupBut it was a .cs , and no makefile (recipe) for how2 build on linux.
And "No Ch.... In He.." i'd let an unknown binary get my pfSense pass./Bingo
-
I have restored many times, with config.xml in the USB root folder. That seems easiest ;)
On full install of 2.6, the USB image includes both a Unix filesystem, and a Windows (FATxx) filesystem labeled pfSense.
By placing config.xml in the latter, it auto-configured during install.
-
@mrpete said in Clarification on using USB stick during installation to restore backup.:
I have restored many times, with config.xml in the USB root folder. That seems easiest ;)
On full install of 2.6, the USB image includes both a Unix filesystem, and a Windows (FATxx) filesystem labeled pfSense.
By placing config.xml in the latter, it auto-configured during install.
It did not work for me. I placed the config.xml file in the root folder of the FAT partition and in /conf/config.xml
What did work for me though is using two USB sticks. One on which is pfSense Image file, and another formated as FAT32 with the config file in the following path: /conf/config.xml. This works during install all the time.
So I think that we have an issue for sure.
The same happens on 2.7.0-development also.
-
I also just ran into this yesterday and intended on posting so thanks for surfacing this thread. I wrote a 3100 image to USB and created \conf\config.xml on the FATRECOV partition. (there was also another partition but Windows says it is unrecognized and offers to format it).
The recover from existing installation didn't work either. "Choose Recover config.xml when the option appears" did not appear that I saw so the installer just overwrite the drive. (which had a valid config file but the OS was broken from a failed upgrade). At what point does that appear? (https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html, "run recovery," choose the (only) eMMC disk, and it finishes)
Luckily I had a recent config file on the USB and had made another copy as well. I hadn't put in a second USB but just restored afterwards. Fortunately I had already disconnected the LAN cable for the office.
Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
Upvote 👍 helpful posts! -
@steveits
I do not have any Netgate hardware, so I cannot test, maybe it's different. I also saw this issue a while back, but I forgot to report it. Maybe if the developers see more comments from us, someone will investigate. At least to let us know if this is a defect or not. -
@steveits said in Clarification on using USB stick during installation to restore backup.:
The recover from existing installation didn't work either. "Choose Recover config.xml when the option appears" did not appear that I saw so the installer just overwrite the drive. (which had a valid config file but the OS was broken from a failed upgrade). At what point does that appear? (https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html, "run recovery," choose the (only) eMMC disk, and it finishes)
The ARM recovery images are not an "installer" in the same way that works for amd64 systems. The ARM recovery images lack config.xml recovery features as they use a completely different mechanism to get the OS onto the target disk.
The ECL method still works there, however, you need to leave the USB drive with the FAT partition plugged in while it boots off its internal disk: https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#restore-using-the-external-configuration-locator-ecl
-
@jimp said in Clarification on using USB stick during installation to restore backup.:
@steveits said in Clarification on using USB stick during installation to restore backup.:
The recover from existing installation didn't work either. "Choose Recover config.xml when the option appears" did not appear that I saw so the installer just overwrite the drive. (which had a valid config file but the OS was broken from a failed upgrade). At what point does that appear? (https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html, "run recovery," choose the (only) eMMC disk, and it finishes)
The ARM recovery images are not an "installer" in the same way that works for amd64 systems. The ARM recovery images lack config.xml recovery features as they use a completely different mechanism to get the OS onto the target disk.
The ECL method still works there, however, you need to leave the USB drive with the FAT partition plugged in while it boots off its internal disk: https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#restore-using-the-external-configuration-locator-ecl
For AMD64 USB memstick images it should work? Or only for ISO images only. Right now in my example it's ignoring the config.xml if it's on the same drive. It works only if the config.xml it's on another separate USB drive.
-
@nrgia It should work from the extra FAT partition on the amd64 installer memstick (VGA, serial, or ADI). The ISO is for optical media or VMs and doesn't have a FAT partition.
-
@jimp said in Clarification on using USB stick during installation to restore backup.:
@nrgia It should work from the extra FAT partition on the amd64 installer memstick (VGA, serial, or ADI). The ISO is for optical media or VMs and doesn't have a FAT partition.
Ok, then it's ignoring the config.xml from the extra FAT partition. I tried it yesterday with pfSense 2.7.0 memstick image.
-
@jimp said in Clarification on using USB stick during installation to restore backup.:
ARM recovery images lack config.xml recovery features
Ah, that explains not seeing the option. Netgate might add a note to that section saying it doesn't apply to ARM. In my case I had backups so it wasn't a big deal, but it sounds like if one has SSH access it's best to grab the config file contents before reinstall.
ECL method still works there, however, you need to leave the USB drive with the FAT partition plugged in while it boots off its internal disk
Hmm, that sounds awfully similar to the "during install" process:
- Boot the install media (Memstick, disc, etc)
- Install to the target disk
- Reboot the firewall
- Remove the USB drive only AFTER the firewall has begun to reboot
So I did leave the stick in place. However this last bullet makes it sound like it does need a second USB stick:
- Remove the install media as well at this point
Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
Upvote 👍 helpful posts! -
@chrcoluk said in Clarification on using USB stick during installation to restore backup.:
Hi
So changed storage in my home pfSense, and I made config backup before I started, made install usb stick using img with rufus, this had a visible fat32 partition with the efi folder inside it.
I made a conf folder and placed the backup config in there named config.xml.
I proceeded to install pfSense with this usb stick as the installer, and on first bootup could see it didnt restore the config.
Documentation here.
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html
I then noticed further down it says you can also restore during any bootup, but using a config folder instead of conf, this one did actually work.
So is it possible its a typo for the during install and should be config for that as well? or did I hit another problem?
From the OP's signature I see that he has a Qotom Q355G4 with an Intel I5. His steps refers to Amd64 boxes, and he refers that he cannot restore the config during the install process, as it's described in the documentation. I followed the same steps, and I confirm that it's not working. So either the code needs to be changed or the documentation.
-
@chrcoluk said in Clarification on using USB stick during installation to restore backup.:
using a config folder instead of conf
Ah, I read right over that. That would explain why the ECL method didn't work for me, and the "during install" with the /conf directory doesn't work on ARM.
-
Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read
config.xml.The correct FAT partition would be labeled
PFSENSEand it would containLICENSE.txtandREADME.txt.I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.
-
@steveits said in Clarification on using USB stick during installation to restore backup.:
@jimp said in Clarification on using USB stick during installation to restore backup.:
ARM recovery images lack config.xml recovery features
ECL method still works there, however, you need to leave the USB drive with the FAT partition plugged in while it boots off its internal disk
Hmm, that sounds awfully similar to the "during install" process:
- Boot the install media (Memstick, disc, etc)
- Install to the target disk
- Reboot the firewall
- Remove the USB drive only AFTER the firewall has begun to reboot
ECL runs on every boot, not just install. It also has slightly different naming requirements, which are outlined in the doc I linked.
So I did leave the stick in place. However this last bullet makes it sound like it does need a second USB stick:
- Remove the install media as well at this point
It works both ways during install, it checks every FAT partition it can find.
-
@jimp said in Clarification on using USB stick during installation to restore backup.:
Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read
config.xml.The correct FAT partition would be labeled
PFSENSEand it would containLICENSE.txtandREADME.txt.I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.
Indeed I used Rufus, and it created another FAT partition containing what you said. I will try again with Etcher. At least nobody will do that mistake again. I will confirm later. Thanks
-
Another possible sticking point is after you write the USB drive you may have to remove it from the system and plug it back in before it sees all of the partitions.
