Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Update from 24.11 to 25.07 failed and possible corrupt system

    Scheduled Pinned Locked Moved General pfSense Questions
    20 Posts 6 Posters 385 Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      steve1515 @stephenw10
      last edited by

      I can, but can I send it privately and also remove keys/passwords?

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by stephenw10

        Yes you can upload it here: https://nc.netgate.com/nextcloud/s/S5y62YEXzXZRbG4

        You can also try to manually run the upgrade yourself if you have another 25.07 install in a VM perhaps.

        Upload the old config file to it (don't import it) then run against it:
        pfSsh.php playback upgradeconfig old_config.xml upgraded_config.xml

        See what errors it produces.

        S 1 Reply Last reply Reply Quote 1
        • S Offline
          steve1515 @stephenw10
          last edited by

          I've uploaded my redacted config. Hope you see something obvious. Thanks!

          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            Hmm, well it upgrades cleanly at the CLI:

            [25.07-RELEASE][admin@plusdev-2.stevew.lan]/tmp: time pfSsh.php playback upgradeconfig /tmp/config-sample.xml /tmp/upgraded_config2.xml
            Upgraded config version level from 23.6 to 24.0
            Done
            0.178u 0.058s 0:00.28 78.5%	3987+349k 0+2io 0pf+0w
            

            And basically instantly.

            Looking through it nothing looks obviously wrong.

            Can I assume it hangs at the config upgrade identically every time you try?

            1 Reply Last reply Reply Quote 0
            • S Offline
              steve1515
              last edited by

              The first time it hung at Checking config backups consistency..., but the 2nd time (after deleting the boot environment and trying the upgrade again) it was hung at Updating configuration........ I'm not sure if those two locations are significantly different or not. I can probably try again tomorrow if it helps?

              If we had to, do you think re-installing using the installer and reloading the config would be a good thing to do? I was thinking about doing that, but also a little worried that it would leave me without my working boot environment to go back to.

              1 Reply Last reply Reply Quote 0
              • A Offline
                abw
                last edited by

                This sounds a lot like what I am experiencing with a 6100 upgrade, also from 24.11 to 25.07. Tried several times, also removing packages, with no success, and the same "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/xmlparse.inc on line 103" error, followed by the rollback to the previous version (love the auto rollback design btw).

                Thanks to information in this thread about "checking backup history", I found that I could produce a timeout-looking error in the UI, by navigating to Diagnostics > Backup & Restore > Configuration History. It would error out after a long delay.
                Turns out that there were a little over 12,500 backup files in the /cf/conf/backup directory, which the UI choked on.
                After copying the backup files, I cleared out all by the last few ("find /cf/conf/backup -name 'config-*.xml' -mtime +7 -delete"), and could now view the backup config history in the UI.

                Tried the upgrade again, and it worked. The upgrade (post reboot) was also way faster.
                Now to figure out why the backup files are building up.

                So maybe try to clear out any old backup files, and see if it helps.

                w0wW E S 3 Replies Last reply Reply Quote 2
                • w0wW Away
                  w0w @abw
                  last edited by

                  @abw
                  Sounds like a bug, did you report it?

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by stephenw10

                    Yes that will work fine and is often a good idea anyway if it's been upgraded for years without reinstalling. It will definitely get you to 25.07.

                    One other thing you could check is the contents of /conf/backup. There was a bug a while ago that created corrupt backup files there that are much too large. Those can cause problems at upgrade.

                    Edit: Oops missed the posts above! Yes check for rogue backup files. This bug: https://redmine.pfsense.org/issues/15994

                    1 Reply Last reply Reply Quote 1
                    • E Offline
                      eloich @abw
                      last edited by

                      @abw said in Update from 24.11 to 25.07 failed and possible corrupt system:

                      /cf/conf/backup

                      Thanks for that! This is exactly what I was experiencing. In my case, the conf history is/was filled with the following:

                      (system): pfBlockerNG: saving DNSBL changes
                      

                      I guess I just have to remember to clean this up on a regular basis.

                      Cheers!

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S Offline
                        stephenw10 Netgate Administrator
                        last edited by

                        It's fixed in 25.07 so you shouldn't have any problems going forward.

                        E 1 Reply Last reply Reply Quote 1
                        • E Offline
                          eloich @stephenw10
                          last edited by

                          @stephenw10
                          Just confirming that the /conf/backup folder is still getting flooded by pfBlocker config backups with 25.07. Happens every hour:

                          Screenshot From 2025-08-06 15-14-07.png

                          Searching through these forums, it's been noted before.

                          Note that while I'm using pfBlocker, I've not activated DNSBL at all.

                          Looked through pfBlocker's config options and can't find a flag that would prevent DNSBL from saving config changes every hour.

                          Anything you need from me/us to investigate further, let me know. Also, if there is a setting that I've missed, please let me know where it is.

                          In the meantime, I've added a task to the pre-work of updating the firewall (i.e. clean up the /conf/backup folder before update).

                          Cheers,

                          E 1 Reply Last reply Reply Quote 0
                          • E Offline
                            eloich @eloich
                            last edited by

                            Replying to myself here...

                            scrolling down, we can see this:

                            Screenshot From 2025-08-06 15-27-46.png

                            I guess I was too lazy to consider scrolling... ๐Ÿ˜‰

                            That would/will fix it, right?

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S Offline
                              stephenw10 Netgate Administrator
                              last edited by

                              pfBlocker updates will still create a new config backup. What has been fixed is the 30 backups limit in the folder. Previously it was possible to end up with a very large number of backups if you did not visit the backup page. And that's what was causing the issue at upgrade when the folder is parsed.

                              E 1 Reply Last reply Reply Quote 1
                              • E Offline
                                eloich @stephenw10
                                last edited by

                                @stephenw10
                                Thanks for confirming! Tried to upvote your post as well as abw's post but unable to at this time since I haven't yet reached a sufficient reputation level.
                                Cheers!

                                1 Reply Last reply Reply Quote 3
                                • S Offline
                                  SteveITS Rebel Alliance @abw
                                  last edited by

                                  @abw said in Update from 24.11 to 25.07 failed and possible corrupt system:

                                  little over 12,500 backup files in the /cf/conf/backup directory

                                  FWIW if using pfBlocker there are two issues. One, at every cron/update run pfBlocker changes an internal timestamp in the config file (thus creating a backup), even if nothing else changed. Two, if using pfBlocker in HA, pfBlocker doesn't sync changes to the secondary properly (without a force update) which can result in double the number of config backups on the secondary. (haven't tried pfBlocker in 25.07 yet)

                                  Viewing the history page should clean out the old files but since that times out it may take a few runs.
                                  ref: https://forum.netgate.com/topic/197685/config-history-not-pruning-on-ha-pair-has-3400-files

                                  If it's problematic for upgrades it might be best if the upgrade process pruned out old files, or checked for "over $backupcount" files and warned.

                                  Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                  When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
                                  Upvote ๐Ÿ‘ helpful posts!

                                  S 1 Reply Last reply Reply Quote 1
                                  • S Offline
                                    steve1515 @SteveITS
                                    last edited by

                                    Just wanted to update everyone. My issue also ended up being the backup config files. Once I deleted the ones older than 7 days, the update went through perfectly.

                                    1 Reply Last reply Reply Quote 1
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.