Update from 24.11 to 25.07 failed and possible corrupt system
-
I can, but can I send it privately and also remove keys/passwords?
-
Yes you can upload it here: https://nc.netgate.com/nextcloud/s/S5y62YEXzXZRbG4
You can also try to manually run the upgrade yourself if you have another 25.07 install in a VM perhaps.
Upload the old config file to it (don't import it) then run against it:
pfSsh.php playback upgradeconfig old_config.xml upgraded_config.xmlSee what errors it produces.
-
I've uploaded my redacted config. Hope you see something obvious. Thanks!
-
Hmm, well it upgrades cleanly at the CLI:
[25.07-RELEASE][admin@plusdev-2.stevew.lan]/tmp: time pfSsh.php playback upgradeconfig /tmp/config-sample.xml /tmp/upgraded_config2.xml Upgraded config version level from 23.6 to 24.0 Done 0.178u 0.058s 0:00.28 78.5% 3987+349k 0+2io 0pf+0wAnd basically instantly.
Looking through it nothing looks obviously wrong.
Can I assume it hangs at the config upgrade identically every time you try?
-
The first time it hung at
Checking config backups consistency..., but the 2nd time (after deleting the boot environment and trying the upgrade again) it was hung atUpdating configuration........ I'm not sure if those two locations are significantly different or not. I can probably try again tomorrow if it helps?If we had to, do you think re-installing using the installer and reloading the config would be a good thing to do? I was thinking about doing that, but also a little worried that it would leave me without my working boot environment to go back to.
-
This sounds a lot like what I am experiencing with a 6100 upgrade, also from 24.11 to 25.07. Tried several times, also removing packages, with no success, and the same "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/xmlparse.inc on line 103" error, followed by the rollback to the previous version (love the auto rollback design btw).
Thanks to information in this thread about "checking backup history", I found that I could produce a timeout-looking error in the UI, by navigating to Diagnostics > Backup & Restore > Configuration History. It would error out after a long delay.
Turns out that there were a little over 12,500 backup files in the /cf/conf/backup directory, which the UI choked on.
After copying the backup files, I cleared out all by the last few ("find /cf/conf/backup -name 'config-*.xml' -mtime +7 -delete"), and could now view the backup config history in the UI.Tried the upgrade again, and it worked. The upgrade (post reboot) was also way faster.
Now to figure out why the backup files are building up.So maybe try to clear out any old backup files, and see if it helps.
-
@abw
Sounds like a bug, did you report it? -
Yes that will work fine and is often a good idea anyway if it's been upgraded for years without reinstalling. It will definitely get you to 25.07.
One other thing you could check is the contents of /conf/backup. There was a bug a while ago that created corrupt backup files there that are much too large. Those can cause problems at upgrade.
Edit: Oops missed the posts above! Yes check for rogue backup files. This bug: https://redmine.pfsense.org/issues/15994
-
@abw said in Update from 24.11 to 25.07 failed and possible corrupt system:
/cf/conf/backup
Thanks for that! This is exactly what I was experiencing. In my case, the conf history is/was filled with the following:
(system): pfBlockerNG: saving DNSBL changesI guess I just have to remember to clean this up on a regular basis.
Cheers!
-
It's fixed in 25.07 so you shouldn't have any problems going forward.
-
@stephenw10
Just confirming that the /conf/backup folder is still getting flooded by pfBlocker config backups with 25.07. Happens every hour:
Searching through these forums, it's been noted before.
Note that while I'm using pfBlocker, I've not activated DNSBL at all.
Looked through pfBlocker's config options and can't find a flag that would prevent DNSBL from saving config changes every hour.
Anything you need from me/us to investigate further, let me know. Also, if there is a setting that I've missed, please let me know where it is.
In the meantime, I've added a task to the pre-work of updating the firewall (i.e. clean up the /conf/backup folder before update).
Cheers,
-
Replying to myself here...
scrolling down, we can see this:
I guess I was too lazy to consider scrolling...
That would/will fix it, right?
-
pfBlocker updates will still create a new config backup. What has been fixed is the 30 backups limit in the folder. Previously it was possible to end up with a very large number of backups if you did not visit the backup page. And that's what was causing the issue at upgrade when the folder is parsed.
-
@stephenw10
Thanks for confirming! Tried to upvote your post as well as abw's post but unable to at this time since I haven't yet reached a sufficient reputation level.
Cheers! -
@abw said in Update from 24.11 to 25.07 failed and possible corrupt system:
little over 12,500 backup files in the /cf/conf/backup directory
FWIW if using pfBlocker there are two issues. One, at every cron/update run pfBlocker changes an internal timestamp in the config file (thus creating a backup), even if nothing else changed. Two, if using pfBlocker in HA, pfBlocker doesn't sync changes to the secondary properly (without a force update) which can result in double the number of config backups on the secondary. (haven't tried pfBlocker in 25.07 yet)
Viewing the history page should clean out the old files but since that times out it may take a few runs.
ref: https://forum.netgate.com/topic/197685/config-history-not-pruning-on-ha-pair-has-3400-filesIf it's problematic for upgrades it might be best if the upgrade process pruned out old files, or checked for "over $backupcount" files and warned.
-
Just wanted to update everyone. My issue also ended up being the backup config files. Once I deleted the ones older than 7 days, the update went through perfectly.
-
Updated to 25.07 today and had the same issue. Hung at updating configuration for about 10 minutes. Reverted back to 24.11, cleared all the pfBlocker config backups (~7k), updated went smooth.
-
@nedyah700 thanks for that.I had the exact same issue and did the same thing as you and then the upgrade succeeded.
-
This post is deleted! -
Thanks. I had the same issue, kept failing boot verification
43000 files in the config backup directory.After getting rid of those, it upgraded faster than I have ever encountered in the past., I had gotten used to upgrades taking 10+ minutes.
