Clarification on using USB stick during installation to restore backup.

SteveITS

@chrcoluk said in Clarification on using USB stick during installation to restore backup.:

using a config folder instead of conf

Ah, I read right over that. That would explain why the ECL method didn't work for me, and the "during install" with the /conf directory doesn't work on ARM.

jimp

Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read config.xml.

The correct FAT partition would be labeled PFSENSE and it would contain LICENSE.txt and README.txt.

I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.

jimp

@steveits said in Clarification on using USB stick during installation to restore backup.:

@jimp said in Clarification on using USB stick during installation to restore backup.:

ARM recovery images lack config.xml recovery features

ECL method still works there, however, you need to leave the USB drive with the FAT partition plugged in while it boots off its internal disk

Hmm, that sounds awfully similar to the "during install" process:

Boot the install media (Memstick, disc, etc)

Install to the target disk

Reboot the firewall

Remove the USB drive only AFTER the firewall has begun to reboot

ECL runs on every boot, not just install. It also has slightly different naming requirements, which are outlined in the doc I linked.

So I did leave the stick in place. However this last bullet makes it sound like it does need a second USB stick:

Remove the install media as well at this point

It works both ways during install, it checks every FAT partition it can find.

NRgia

@jimp said in Clarification on using USB stick during installation to restore backup.:

Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read config.xml.

The correct FAT partition would be labeled PFSENSE and it would contain LICENSE.txt and README.txt.

I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.

Indeed I used Rufus, and it created another FAT partition containing what you said. I will try again with Etcher. At least nobody will do that mistake again. I will confirm later. Thanks

jimp

Another possible sticking point is after you write the USB drive you may have to remove it from the system and plug it back in before it sees all of the partitions.

NRgia

@jimp said in Clarification on using USB stick during installation to restore backup.:

Another possible sticking point is after you write the USB drive you may have to remove it from the system and plug it back in before it sees all of the partitions.

If you where responding to me I don't understand what you meant. After the FAT partition is also created, I copy the config.xml file to that partition. So if I can copy something to it, it sees it. Also from the boot manager I can see 2 partitions, and I can choose either of them. Normally it only from one of them. But the information is there. I will try with Etcher instead.

jimp

On some operating systems it won't refresh the disk partitions after writing, depending on how it was written. Unplug the USB drive and plug it back in and you will know for certain it's re-reading the partition table properly. You might think you are seeing the right one, but the list may not be complete.

NRgia

@jimp said in Clarification on using USB stick during installation to restore backup.:

On some operating systems it won't refresh the disk partitions after writing, depending on how it was written. Unplug the USB drive and plug it back in and you will know for certain it's re-reading the partition table properly. You might think you are seeing the right one, but the list may not be complete.

Ok, just tried it.

OS: Manjaro Linux (previously Win 10)
Flashing tool: balenaEtcher v.1.7.7 (previously Rufus)

I unmounted and remounted the USB stick.
Created a conf directory where I copied the config.xml
Booted the pfSense box from USB installation media.
Installed pfSense.
Selected Reboot at the end of the installation.
During POST I removed the installation media, otherwise we have the ECL case.

Actual: No configuration has been restored, and pfSense asks me to reconfigure the Networks, define which interface is WAN, LAN, the user/password is set to default.

Expected:
The user configuration from /conf/config.xml should be restored during installation phase.

MrPete

@nrgia said in Clarification on using USB stick during installation to restore backup.:

Ok, just tried it.
...
6) During POST I removed the installation media, otherwise we have the ECL case.

Actual: No configuration has been restored, and pfSense asks me to reconfigure the Networks, define which interface is WAN, LAN, the user/password is set to default.

Expected:
The user configuration from /conf/config.xml should be restored during installation phase.

So, this IS a point of at least confusion or uncertainty.

@jimp QUESTION: are we SURE that autoconfig-during-install is NOT at all dependent on the ECL process?

Here's how I do my installs:

Set system to boot from internal drive if possible
Do a one-time boot from install USB (or if not available, wipe the internal boot drive)
Do the USB install
Leave USB stick in for at least the first reboot.

Even the ECL process link shared by @jimp says "If this is the first boot post-installation, then this process also triggers reinstallation of packages listed in the restored configuration."

I suspect pfSense does NOT self-reconfigure based on reading config.xml during the install-time boot, but instead during the next boot. But I'm not certain of that.

SteveITS

@mrpete Do you copy the config file to both /conf/config.xml and /config/config.xml? The two procedures use different locations.

stephenw10

I've done this many times and I expect it to boot into the recovered config. You should not have to reboot again unless you had a package that required it after install.

Steve

noplan

@stephenw10

OK gonna test it tomorrow cuz I am not able to remember if I disconnected the stick after the first reboot....

Da mm n getting old sucks....

chrcoluk

@jimp said in Clarification on using USB stick during installation to restore backup.:

Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read config.xml.

The correct FAT partition would be labeled PFSENSE and it would contain LICENSE.txt and README.txt.

I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.

I can confirm I am using AMD64.

I will be installing 2.6.0 soon so I get the new ZFS layout, I will write the image with Etcher when I do this and report back. :)

MrPete

@steveits Actually I only place it in the root these days.

chrcoluk

@jimp said in Clarification on using USB stick during installation to restore backup.:

Do not use Rufus, use Etcher. The EFI partition is NOT the FAT partition used to read config.xml.

The correct FAT partition would be labeled PFSENSE and it would contain LICENSE.txt and README.txt.

I just wrote out a 2.6.0 memstick with Etcher and it had the correct partition present.

Hi Jim, mystery solved, Seems the partition is created but a drive letter isnt been assigned to it so isnt visible in windows. So I guess yes a Rufus problem, but fixable by manually adding a drive letter so can copy the files across, see this screenshot. Feel free to use this image if you want to edit the documentation.

Although I am wring the image now with Etcher.

alt text

Confirmed using Etcher both FAT32 partitions show up in windows with drive letters automatically, copied over the config to the PFSENSE partition and it worked during install as documented. I suggest adding something to the documentation about what was said in this thread. :)

I will email the Rufus developer about this issue.

jimp

We have removed all of the Rufus information from the docs (too problematic) so I'm hesitant to add anything back even if there is a workaround. Using Etcher is so much simpler and less likely to cause problems. There isn't a need to involve something as complex as Rufus here.

Rufus can make sense for Linux and for other similar situations where you need more control over how the disk is laid out and Linux boots, but not for pfSense/FreeBSD.

scurrier

Seems like this is still a problem. I tried both Etcher and Rufus on Windows. They both appear to have done the same thing, creating EFISYS (FAT), DTBFAT0 (FAT32), and some other partition I don't know much about from Windows Disk Manager. It's just called "USB Drive" but with no ability to show properties. It automatically mounted, but I can't open it. I tried disconnecting/reconnecting the USB with each flash software - no change.

I have to do this on the road tomorrow on our 1100 so I'm just going to spam the files in all the possible locations on each usable partition and hope for the best. Hope I don't need to use the network to restore because I'm under the gun for time.

stephenw10

The aarch64 image is different. It is the DTBFAT0 partition you should put the config.xml file in, you will see the 4 dtb files in there when you open it.
However the aarch64 installer does not copy that in during the install. Instead it will pull it in at the first boot after the install using the External Config loader:
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#restore-using-the-external-configuration-locator-ecl

Steve

scurrier

@stephenw10 thanks, I have that one covered so should be good to go. Nice to have some comfort about it. I'll report back how it goes.

scurrier

@scurrier it went pretty well. Only things that threw me off were:

The serial console was somehow cutting off some of the text at the login screen before rebooting, even though I had already configured my terminal correctly in the last, leading me to think my settings were wrong. They weren't, it was some pfsense console problem fixed by rebooting.
The password prompt for the encrypted config has a timeout but doesn't say so, so when I went to my password manager to get the password, it timed out and I had to reboot and try again.