Method to save configuration after fatal trap / panic

HydeTech · Nov 8, 2020, 9:12 AM

Our SG-5100 seems to have crashed. In another stroke of bad luck, I can't find a backup. Below, is how far it boots before stopping. Is there ANY possible way to retrieve the settings so I am not starting 100% from scratch? Netgate tech support is telling me I need to re-image. I'm dreading the many hours it's going to take to get this Firewall back to where I had it if I do that without being able to extract the settings.

Alternate question: Any way around having to re-image based on where it's halting?

I am able to interrupt the load of pfSense and get to the loader prompt. From this prompt, I can see the file system. Is there any documented method of saving off pertinent settings from the pfSense loader prompt or from the kernel?

If I allow it to attempt a full load, it gets to this point and halts:

mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
Fatal trap 1: privileged instruction fault while in kernel mode
cpuid = 0; apic id = 04
instruction pointer = 0x20:0xffffffff830c0a54
stack pointer = 0x28:0xffffffff830c0a48
frame pointer = 0x28:0xffffffff830c0a40
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (swapper)
trap number = 1
panic: privileged instruction fault
cpuid = 0
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at kdb_enter+0x3b: movq $0,kdb_why

HydeTech · Nov 8, 2020, 9:26 AM

Is this what I am looking for? If so, how do I get them off the netgate and stored in a safe place?

/cf/conf/backup/config-*.xml

Are each of these a full backup of a prior state or do I need all of them?

If I'm not on the right track, please let me know.

heper · Nov 8, 2020, 9:29 AM

@HydeTech

Each is a full state... You only need 1 of them.

Is there a usb port available on the sg device?

HydeTech · Nov 8, 2020, 9:41 AM

@heper Yes. There are two USB ports available. I'm just a complete amateur at using the console on this device. Would you mind pointing me in the right direction on how to mount a USB stick AND how to copy the files over to it? I know that sounds like Amateur Hour 101, but I'm fumbling around here. What format do I need the USB stick to be in for the Netgate to recognize it? (I'm a bit shocked that official support channels told me I had to re-image and that I was out of luck on my backups. Glad I didn't believe them and started digging). Thanks for your help.

Alternatively, can I just restore from the loader prompt or kernel somehow?

heper · Nov 8, 2020, 9:55 AM

i don't own an sg5100... so this might not be accurate

format a usb drive with fat32 on a pc/laptop
insert usb drive in sg5100
on command line type

[2.4.5-RELEASE][root@pfSense.lan]/root: dmesg

at the end of the screen-output you should see something like this:

da1 at umass-sim0 bus 0 scbus4 target 0 lun 0
da1: <USB_STICK_BRAND 1.04> Fixed Direct Access SCSI-4 device
da1: Serial Number WD-WXE508CAN263
da1: 40.000MB/s transfers
da1: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
da1: quirks=0x2<NO_6_BYTE>

this can be da0,da1,da2,da3,....

when you found your device node above & verified is is indeed your usb drive, then you can proceed to mounting it:

[2.4.5-RELEASE][root@pfSense.lan]/root: mount_msdosfs /dev/da1s1 /mnt/

after mounting is succesful, you can copy de config

[2.4.5-RELEASE][root@pfSense.lan]/root: cp /cf/conf/backup/config-kdjkldjfkl.xml /mnt/

after that you need to unmount the usb drive.

[2.4.5-RELEASE][root@pfSense.lan]/root: umount /mnt

after above is complete, it is safe to remove the usb-drive & store the config somewhere safe on an other pc

heper · Nov 8, 2020, 9:54 AM

@heper said in Method to save configuration after fatal trap / panic:

@HydeTech
(I'm a bit shocked that official support channels told me I had to re-image and that I was out of luck on my backups. Glad I didn't believe them and started digging).

at the start you claimed that the device didn't boot. now it appears you've managed to get to the console.
I'm sure the netgate staff would have assisted you in recovering the config if you told them you managed to get through the boot into a working consoleshell

HydeTech · Nov 8, 2020, 10:00 AM

@heper said in Method to save configuration after fatal trap / panic:

i don't own an sg5100... so this might not be accurate

format a usb drive with fat32 on a pc/laptop

insert usb drive in sg5100

on command line type
[2.4.5-RELEASE][root@pfSense.lan]/root: dmesg
at the end of the screen-output you should see something like this:
da1 at umass-sim0 bus 0 scbus4 target 0 lun 0
da1: <USB_STICK_BRAND 1.04> Fixed Direct Access SCSI-4 device
da1: Serial Number WD-WXE508CAN263
da1: 40.000MB/s transfers
da1: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
da1: quirks=0x2<NO_6_BYTE>
this can be da0,da1,da2,da3,....

when you found your device node above & verified is is indeed your usb drive, then you can proceed to mounting it:
[2.4.5-RELEASE][root@pfSense.lan]/root: mount_msdosfs /dev/da1s1 /mnt/
after mounting is succesful, you can copy de config
[2.4.5-RELEASE][root@pfSense.lan]/root: cp /cf/conf/backup/config-kdjkldjfkl.xml /mnt/
after that you need to unmount the usb drive.
[2.4.5-RELEASE][root@pfSense.lan]/root: umount /mnt
after above is complete, it is safe to remove the usb-drive & store the config somewhere safe on an other pc

How did you get here in the first place? I seem to be in a completely different system. Some kind of pfsense loader command prompt with very very few commands compared to a linux shell (which is what it appears you are in).

Note that I have no IP based connectivity. I'm using the console USB connection on a COM port.

heper · Nov 8, 2020, 10:02 AM

i can't know what you are seeing on your screen ....

HydeTech · Nov 8, 2020, 10:20 AM

@heper said in Method to save configuration after fatal trap / panic:

i can't know what you are seeing on your screen ....

Sorry. I appreciate your help. Here's what I am seeing. This is a "pfSense loader prompt" that does not appear to be a linux shell. I can't seem to figure out how to get to a linux shell. It seems to be going right into booting pfSense and then I'm stuck there.

I wonder if I could create a linux boot USB stick and then mount the internal drive after booting with the USB based Linux.
Seems like I should be able to get to the Linux kernel that's alraedy on the Netgate, but I haven't figured out how so far.

Here's where I am.

HydeTech · Nov 8, 2020, 10:37 AM

@heper Unless you have an easier suggestion, I think I'm going to try creating a Ubuntu boot USB stick and try to retrieve the backup file(s) that way. I do need some sleep and so I'm signing off for now. If you have any additional thoughts I would definitely be appreciative. I will not be replying again for at least a few hours. I've had it for the night and we do have a backup firewall of sorts in place to keep us hobbling along until I get this resolved (an old Unifi security gateway). Thanks again.

heper · Nov 8, 2020, 7:14 PM

@HydeTech said in Method to save configuration after fatal trap / panic:

you have any additional thoughts I would definitely be appreciative. I will not be replying again for at least a few hours. I've had it for the night and we do have a backup firewall of sorts in place to keep us hobbling along until I get this resolved (an old Unifi security

thats the bootloader ...i don't think it is possible to recover a config from within the bootloader.

you tried booting into single user mode right ?

it might be possible to boot an ubuntu usb drive, but i doubt it'll be straightforward or easy to do

HydeTech · Nov 8, 2020, 7:14 PM

@heper said in Method to save configuration after fatal trap / panic:

@HydeTech said in Method to save configuration after fatal trap / panic:

you have any additional thoughts I would definitely be appreciative. I will not be replying again for at least a few hours. I've had it for the night and we do have a backup firewall of sorts in place to keep us hobbling along until I get this resolved (an old Unifi security

thats the bootloader ...i don't think it is possible to recover a config from within the bootloader.

you tried booting into single user mode right ?

it might be possible to boot an ubuntu usb drive, but i doubt it'll be straightforward or easy to do

I did try booting single user mode. It fails too. Wish me luck on what I'm about to start trying, which is the Ubuntu approach. I'll post my results, and if I'm successful, I'll post the steps I took so if someone else finds themselves in this predicament, maybe it will help them at some point in the future.

stephenw10 · Nov 9, 2020, 6:47 PM

You can recover the config from the pfSense install image as long as the partition is not completely destroyed, it gives you option before you install:
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#recover-config-xml-from-existing-installation

Worst case you can cat the recovered config to the console from there and copy it out into a file.

Steve