Recover config file from ISO during (re)install?

rgijsen

I can't properly copy / paste as booting from ISO doesn't give me network, and I don't have serial on Hyper-V. So I'll try like this:

During boot I don't see the ECL logging anything at all, even with verbose enabled. Or it's too quick for me to see. What log file would that be in? I attached /var/log/messages, but after recognizing the disks, I don't see it referenced again. Note that I actually ran the config-recovery option before taking the log file. My apologies if I'm misunderstanding.

messages.txt

stephenw10

Ok great. And in the conf directory you have config.xml?

The ECL runs during a normal boot of the installed pfSense not booting the installer. But it looks in the same places as the config recovery that runs in the installer so if it appears there I'd expect it to also recover from there.

rgijsen

@stephenw10 said in Recover config file from ISO during (re)install?:

Ok great. And in the conf directory you have config.xml?

The ECL runs during a normal boot of the installed pfSense not booting the installer. But it looks in the same places as the config recovery that runs in the installer so if it appears there I'd expect it to also recover from there.

yes, config.xml is in /conf.

stephenw10

Ok alternative approach!

Drop to the rescue shell from the installer then run the config recovery manually from there like:
/root/recover_configxml.sh

That will then show exactly what it's doing and you'll be able to see it.

rgijsen

That simply shows:

No suitable disk partitions found

whether I mounted the partition or not.

stephenw10

Hmm, Ok I guess that only looks for existing installs. Let me see....

stephenw10

Yup. OK try:
/usr/libexec/bsdinstall/copy_configxml_from_usb

It should check anything that shows up as fat32 in gpart show -p so that should include da1p2.

rgijsen

@stephenw10

It shows as ms-basic-data, I'm not sure what it should say. But it's formatted as FAT32 in Windows, and BSD can surely read it.

stephenw10

Ok that's some windows formatting problem then. It's not marking it as fat32. It should show like:

[23.09.1-RELEASE][root@8200.stevew.lan]/root: gpart show -p
=>       40  234441568    nda0  GPT  (112G)
         40     532480  nda0p1  efi  (260M)
     532520       1024  nda0p2  freebsd-boot  (512K)
     533544        984          - free -  (492K)
     534528    2097152  nda0p3  freebsd-swap  (1.0G)
    2631680  231809024  nda0p4  freebsd-zfs  (111G)
  234440704        904          - free -  (452K)

=>      1  1978367    da0  MBR  (966M)
        1    66584  da0s1  efi  (33M)
    66585  1727568  da0s2  freebsd  [active]  (844M)
  1794153   131072  da0s3  fat32  (64M)
  1925225    53143         - free -  (26M)

=>      0  1727568   da0s2  BSD  (844M)
        0       16          - free -  (8.0K)
       16  1727552  da0s2a  freebsd-ufs  (844M)

jimp

That's a Windows partitioning issue and unfortunately it's not safe to assume anything based on it. Partitions with that type could be NTFS, ExFAT, FAT32, whatever, so we can't just add code to allow that type alongside the current code.

https://en.wikipedia.org/wiki/Microsoft_basic_data_partition

You'll need to find a way to change that partition type. Some googling suggests it can be done but it's not exactly trivial.

rgijsen

hmm weird. Even when I create a FAT partition (ie. FAT16) it's reported as ms-basic data. We have minitool partition wizard, which is a bit like GParted for Windows, very extensive, but when I create FAT or FAT32 with that, it's still the same. As I think the recovery console misses mkfs, I booted gparted live, and THAT actually saw my 'MS FAT16' partition as FAT16:

So I created a new 1GB FAT32 partition using GParted:

And that still shows up as ms-basic-data

So as last test I booted an old laptop with pfsense 2.7.2 installer, and inserted another USB stick. That is actually reported as fat32lba (opposed to just fat32 in your reply). But the FAT32 partitions on the internal SSD (Windows 10) also report as ms-basic-data.

What I think is weird is that the gparted created partitions are still reported as ms-basic-data. Could that be a BSD thing on harddisks / SSD maybe? As mount_msdosfs can actually read them fine as well, so they do seem to be valid partitions.

Anyway, if the way BSD identifies partition changed, maybe an update would be a nice to have. If not, I wouldn't bother honestly, as it's a bit of a edge-case anyway. If recovery from ISO is planned though, that requires scanning ISO9600 filesystem, so maybe at the same time ms-basic-data could be added as well. But I haven't looked into the code, and I leave that descission to you ;-)

If there's anything else I can test, or create a partition with from pfSense recovery shell itself, let me know and I'll test.

Thanks so far!

stephenw10

It should still match fat32lba:
https://github.com/pfsense/FreeBSD-src/blob/devel-main/usr.sbin/bsdinstall/scripts/copy_configxml_from_usb#L40

The disk I showed there is just the pfSense installation image. So you could try adding that as an attached disk.

rgijsen

Oh I'm sure it would've found the config on the USB stick on that laptop, as that was indeed seen as fat32. Didn't even try that, as gpart showed FAT32 there.

For test with my vhdx, I modded line 40 in copy_configxml_from_usb to:

for try_device in `/sbin/gpart show -p | /usr/bin/egrep '(fat32|\!11|\!12|\!14)|ms-basic-data' | /usr/bin/awk '{print $3;}'`; do

and then it works just fine, just as with an USB stick. As the ms-basic-data seems to be how BSD reports these valid FAT/FAT32 drives on at least Hyper-V disks, even when created from Linux / gparted, it might be nice to have that functionality. For some weird edge-case like this, where someone is re-installing a virtualized pfSense without access to USB and no ISO fetching yet, I actually don't see an issue adding that line in the distribution, and it might just save someone some time in a disaster.

stephenw10

Whilst it still could be any partition type inside that it will fail to mount anything but FAT types. And that shouldn't be an issue given the code there. Needs more review though.

justme2

FWIW - this also fails to work on a USB "install" with the config.xml file located under /conf on the DOS partition of the USB install drive. In this case its going from 2.7.2 to 2.7.2 - so should be straight forward. There are a vast number of lines that fly by stating something about restore option(s) not specified and then truncates the config.xml file on the freshly installed system to 0 bytes. Would be nice if it would retain a copy of the ORIGINAL file in the /conf/ directory for such occasions. Without a doubt, something isn't right. Had to manually mount the USB drive (physical firewall - not VM), copy the file over to /conf/ and then reboot. Then it seemed to work as expected. Seems to infer that someone may have missed validating this function prior to release. A bit of a PITA, but you can get around it.

eg:
gpart show
(look for daN device with DOS partition)
mount -t msdosfs /dev/(USB DOS PARTITION ID) /mnt
copy /mnt/conf/config.xml /conf/
reboot
(pull the USB device)

Only realized this issue when switching from a single device to a ZFS mirror on one device. At first it was a <soil linen moment> for a couple minutes while trying to figure out what happened. Thought the install media was bad, so created another installer on fresh media - no go either. It wasn't until the second go round where more of the boot messages were caught and wondered if it wasn't just "copy, reboot and be done with it". Which turned out to be the case. Just script a good solid automated backup and make sure that you periodically check to ensure that they are being captured "somewhere safe". Would like to have seen this "just work" as that inspires confidence in the product vs. being concerned (potentially leery of reliability from what appears as a "failed" restore) that something "might not be right" and could become a problem later. Thankfully, still have the prior device so that it could be swapped back in, if something seems problematic within the next couple of days.

Can only imagine a VM being a bit more problematic if you can't readily synthesize a USB drive with the config on it for mounting. Perhaps create a small 1G "disk" in the virtual environment, layout a filesystem and have a copy of the config.xml file as a "ready to go" option. If you run into this again, shutdown the VM, attach the 'disk' and then after boot, mount the drive to gain access to the config. (Hindsight is always 20/20....)

rgijsen

[quote]
Perhaps create a small 1G "disk" in the virtual environment, layout a filesystem and have a copy of the config.xml file as a "ready to go" option.
[/quote]

That's exactly what I wanted to do here. Although planned, so I had all the time to figure things out and copy the config file to the correct location myself. But yes, in case of true disaster and both HA nodes fail or whatever, it would be nice be up and running in like two minutes rather than 10 when you manually need to configure network before being able to access it. Which would still be fast but still.

stephenw10

@justme2 said in Recover config file from ISO during (re)install?:

There are a vast number of lines that fly by stating something about restore option(s) not specified and then truncates the config.xml file on the freshly installed system to 0 bytes.

You have any logs from that? That function has worked well for years I'm not sure what could have changed there.

Steve

justme2

@stephenw10

Grep'ing through the logs and apparently whatever process does the restore - does not write any output directly (or via syslog) to anything under /var/log. But on the console (physical monitor/keyboard) there was more one than one "screen's worth". It was a message about ?4? different options and at least one had to be specified in order to perform the restore. ie: At least one of a, b, c or d must be specified type of message. I recall "x" being one of the options. Thus, whatever process actually performs the restore is being called without a parameter and failing. Perhaps if the output from the restore process had been captured in the system log/s (as the restore/recover occurs as part of the first reboot after install, thus it should be possible to capture on the installed storage) - it would be possible to provide that information.

stephenw10

Hmm, you can try running it manually if you drop to the recovery shell from the installer and run:
/usr/libexec/bsdinstall/copy_configxml_from_usb

That should show you any errors that it hits.

justme2

@stephenw10

Would need to schedule another change control (as it impacts prod services) in order to test - not sure that it would be approved until sometime after the middle of January.

Looking at the installed system, there is nothing in that directory. Any reason that the install wouldn't carry that over? (Checked a couple other firewalls that were 'upgraded' from prior versions - not "fresh" installs and they lack anything in that directory as well). If one had the script to actually review, could probably find the error that comes back (if it's in that file).