pfSense 2.4.3 configuration lost after lack of disk space - how to recover?
-
Hello folks.
We have a pfSense 2.4.3p, running on Citrix XenCenter.
The VM was created years ago, and was set with only 10GB of space for the disk.
So, today, after trying to login in the WebUI i started getting a HTTP_REFERER errors.
Then, the OpenVPN died, and we lost connection to the pfSense, since its on a remote site.After gaining some access via screen-sharing and getting on the console in XenCenter - we see the following:
Perhaps a bug. Still, there were plenty of messages about "no free disk space left on X".
First thing to check - logs. No logs were clogging up the space.
Almost everything was being eaten by .db files.So i went off with a reboot, with the idea to have some temporary files to be freed up automatically. That did not help.
In fact, it made things worse.
Somehow, the network configuration went kaboom. Users in the office lost DHCP. The pfSense had 1 WAN and 1 LAN address set up. Seems like, when trying to reboot, the pfSense tried to upgrade packages... which seems bonkers, since there is no free space left. And somehow it nuked the network configuration.I went up with expanding the VM disk, deleting the Swap partition, setting the fstab entry for the root partition to use the block device instead UUID. Then grow the FS.
Reboot again -> nada. All configuration still missing.Try to restore from backup - there are 15 new backups from today for no reason at all (from the package installation).
I restore to a configuration from 2 months ago.
Reboot -> still no network configuration.
Perhaps backup was corrupted.
Check the backups again - all backups are from today now. Because it installs automatically updated packages.
With none having network configuration in it.I have tried setting up the network again - xn1 for WAN, with the public address, on a /24, with the upstream gateway.
pfSense now can ping the outside world.
But it cant be reached from outside.
Then setup the LAN on xn2, with the previous set IP address and a /24.
It cant be reached from the LAN for some reason.DHCP is still not working. Users in the LAN need to setup their IPs manually, so they can have connectivity (we have 2 ISPs, and 1 more gateway i think).
All data seems to be still present on the OS, but somehow the pfSense configuration is gone.
Any ideas and steps on how to recover from this?
PS: No backups outside of the VM are available, because... someone did not set them up.
-
Do you have a VM snap you can roll back to?
Do you have any config backup at all?
If not then I would just reinstall and recreate it and use a current version. 2.4.3 is ancient!
Steve
-
@stephenw10
Found ancient configuration from 2018, but that would require alot of users to be recreated.Since this is configuration only, and there are .db files, is the data not actually being held inside those files?
But yea, if there is no other option, i would guess we will import back from that old configuration and then initiate an upgrade to something more recent (if thats even possible).
Going from the ground to create everything again is going to be a royal PITA, since this is remote site, which is also in production... -
Anything that may have been present would have replaced at boot.
Any backup config files would be in /conf/backup. There is also /tmp/config.cache but that will just be the current config.
If you had Auto Config Backup configured that would have 100 backups so should still have a valid one.
You can restore a 2.4.3 config into 2.6 no problem.
If the only significant change is the users list you can probably manually insert it.
Steve
-
@stephenw10 The Auto backup was set to 30, and at reboot it seems to do a package upgrade... which overwrites those backups.
What do you mean, that anything present would be replaced at boot? -
For example the running firewall users are created from the config file at each boot. So whatever you have on there now is the result of the config that was present at the last boot.
