Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    updating to acme 1.0 breaks system beyond repair: need to restore from backup

    Scheduled Pinned Locked Moved ACME
    25 Posts 10 Posters 1.9k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Offline
      MarvinFS @jwt
      last edited by MarvinFS

      @jwt said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

      installed or upgraded?

      2.7.2 was installed first from iso then upgraded to 2.8 then config restored - that resolved the issue.

      @jwt said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

      more details here would be good.

      no details - unfortunately installer doesn't provide any logs or troubleshooting info, except "unable to contact netgate servers" during install. Despite same PPPoE username and password specified in the initial prompt as used during normal operations.

      1 Reply Last reply Reply Quote 0
      • R Offline
        raidflex
        last edited by raidflex

        Same issue, completely broke my system as soon as I started the package upgrade. I could not even SSH into the system and upon rebooting the system it would not boot anymore. I had to also restore from a backup. I have never had a package in pfsense completely hose the system like that.

        When I ran the reinstall of 2.8.0 l restored the config directly from the drive and everything is working. The ACME package is also on version 1.0 after the restore now.

        S 1 Reply Last reply Reply Quote 0
        • S Offline
          slu @raidflex
          last edited by

          I upgrade from 0.9 to 1.0 on 2.8.0 today, no issue.

          pfSense Gold subscription

          R 1 Reply Last reply Reply Quote 0
          • R Offline
            raidflex @slu
            last edited by

            @slu said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

            I upgrade from 0.9 to 1.0 on 2.8.0 today, no issue.

            There must be some particular config or package conflict that causes the issue for some people. A package should not be able to break the system that bad though.

            1 Reply Last reply Reply Quote 0
            • jimpJ Offline
              jimp Rebel Alliance Developer Netgate
              last edited by

              The OP of this thread had unsupported packages installed from third-party repositories, which is almost certainly where the problem originated. It doesn't matter if the repo configs were removed -- if local packages were installed/changed by those repos, those packages are still present.

              While we do not deliberately break such configurations, if you install a package from unsupported repositories and they replace or mess with base system dependencies, then there is no telling what will break over time like this.

              One of many reasons we discourage using anything but Netgate repositories for packages.

              Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              M 1 Reply Last reply Reply Quote 1
              • M Offline
                MarvinFS @jimp
                last edited by

                @jimp said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

                While we do not deliberately break such configurations, if you install a package from unsupported repositories and they replace or mess with base system dependencies, then there is no telling what will break over time like this.

                Understood - thank you very much for the clarification. I need crowdsec though... and there are no official support yet. I don't mind reinstalling the system, it takes reasonable amount of time, unless I found netinstaller fails to connect to my pppoe which tripled the time of restoration. For that I have no explanation and it is obviously not related to the dependencies, but that's offtopic in this thread.

                1 Reply Last reply Reply Quote 0
                • M Offline
                  MarvinFS
                  last edited by MarvinFS

                  OK I've done some additional testing, as I was curios what the hell is happening - so I can now confirm installing crowdsec with their packages breaks further packages installation system completely.. regardless cli or UI.. not sure about 2.7.2 but 2.8.0 got completely broken
                  I used that:
                  https://docs.crowdsec.net/docs/next/getting_started/install_crowdsec_pfsense/

                  Then I've also tried to install qemu-guest tools and to reinstall ACME - the very same situation with REMOVE packages... so crowdsec breaks the system I believe. I'm too lazy to report to their github, I guess I'll just avoid them until official support from Netgate.
                  @jimp

                  M 1 Reply Last reply Reply Quote 0
                  • M Offline
                    MarvinFS @MarvinFS
                    last edited by

                    on the other hand I might report it...

                    R 1 Reply Last reply Reply Quote 0
                    • R Offline
                      raidflex @MarvinFS
                      last edited by raidflex

                      @MarvinFS said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

                      on the other hand I might report it...

                      It's been years since the Crowdsec package was first released for pfSense and it has yet to be added as an official package, not sure why. So prob better to report it because who knows if it ever will be released.

                      I waited a while for the official package before installing Crowdsec, but gave up and have been using it for at least 6 months without issue until the ACME update. I try not to install anything outside the official packages, it's the only one that I have done this way. I will just have to be more careful with updates, and maybe uninstall Crowdsec when applying other updates first.

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        MarvinFS @raidflex
                        last edited by

                        @raidflex said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

                        maybe uninstall Crowdsec when applying other updates first.

                        It seems like it doesn't help at least from what I see on my system... it changes something.. so it must be definitely reported to their github. I have never experienced that before and crowsec was installed.. maybe with 2.8.0 something have changed

                        1 Reply Last reply Reply Quote 0
                        • X Offline
                          Ximix
                          last edited by

                          I also crashed my system yesterday with an ACME update. I can't say whether I had Crowdsec installed or not. But I've tried things like that in the past, so it's quite possible.

                          1 Reply Last reply Reply Quote 0
                          • P Offline
                            proven3257
                            last edited by proven3257

                            I also had a fatal crash yesterday. My fix after restoring from a backup was to uninstall crowdsec using the install script sh install-crowdsec.sh --uninstall, updating and installing additional packages, and then fetching the new install script as shown on the crowdsec site. This seems to allow everything to work without needing to reconfigure anything and I am doing some testing to ensure that crowdsec is actually working. After installing pfblocker, it seems that re2 needed to be removed and was the likely cause of the conflict.

                            I installed crowdsec a while back so I don't know if the issue is fixed but the latest version of the install script mentions breaking dependencies and what seems to be a new baked in fix for it (Versions that are distributed with dynamically linked abseil/re2 are not recommended to avoid dependency issues), and installing other packages with crowdsec currently installed does not seem to break things now.

                            M 1 Reply Last reply Reply Quote 0
                            • M Offline
                              MarvinFS @proven3257
                              last edited by MarvinFS

                              Yes seems like they release a fixed script and new version doesn't break anything! I've tried on my system already!

                              v0.1.5 w/ crowdsec 1.6.11 and 0.0.33 (with support for pfsense 2.8) Latest
                              @mmetc mmetc released this 20 Aug 15:01
                              v0.1.5-1.6.11-33-2.8
                              This is the first version tested with pfsense 2.8.
                              The new install script will also fix a dependency issue caused by the previous versions.

                              1 Reply Last reply Reply Quote 0
                              • provelsP Offline
                                provels
                                last edited by

                                This same mess happened to me, even w/o Acme, going from 25.07 to *.1. Blew, reinstalled w/ Crowdsec, blew again, reinstalled, clipped all the Crowdsec info from config.xml, restored config, back to normal. Crowdsec is a great concept, but I think I'm out.

                                Peder

                                MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                                BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                                R 1 Reply Last reply Reply Quote 0
                                • R Offline
                                  raidflex @provels
                                  last edited by

                                  @provels said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

                                  This same mess happened to me, even w/o Acme, going from 25.07 to *.1. Blew, reinstalled w/ Crowdsec, blew again, reinstalled, clipped all the Crowdsec info from config.xml, restored config, back to normal. Crowdsec is a great concept, but I think I'm out.

                                  I never had this issue with Crowdec before the ACME update, even with updating from 2.7 to 2.8 there was no issues. In fact after restoring from a backup after the ACME update, Crowdsec reinstalled just fine, and this was before the recent release a couple days ago that contained a fix.

                                  JeGrJ 1 Reply Last reply Reply Quote 0
                                  • JeGrJ Offline
                                    JeGr LAYER 8 Moderator @raidflex
                                    last edited by

                                    @raidflex said in updating to acme 1.0 breaks system beyond repair: need to restore from backup:

                                    In fact after restoring from a backup after the ACME update, Crowdsec reinstalled just fine, and this was before the recent release a couple days ago that contained a fix.

                                    Yeah, that may be, but if you install packages with dependencies on the console rather then the package manager, those may have (old) dependencies for specific versions. So if that crowdsec package has a dependency on an older pfsense base package or something like that and you install any other package (like Acme) which may collide with that, the package manager makes a decision to solve the conflict. Not always the most sane one - sure - but that's like any other distro out there. Manually installing packages on the console always may get you into dependency hell :)

                                    Just saying, because now it was acme, next time it could easily be some other package triggering such an effect.

                                    Cheers

                                    Don't forget to upvote πŸ‘ those who kindly offered their time and brainpower to help you!

                                    If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.