2.4.5R1 Update no package re-install

Visseroth

I just updated to 2.4.5R1 and noticed the packages didn't reinstall so I tried to do some updates on some that said they needed to be updated and no go so I went ahead just for the heck of it did a "Reinstall All" and it is stuck on "Please wait while the update system initializes"
Suggestions?

Edit: I also tried pkg update && pkg upgrade from the command line and it reported... "pkg: Cannot get an advisory lock on a database, it is locked by another process"

revengineer

I found the package reinstall process got stuck. To see if you suffer from this, check with "ps aux|grep pkg-install". If such processes are found, kill them with "killall pkg-static". You may have to do this multiple times. I found that the packages are actually installed, so just make sure you give pkg-static time to install the package before killing it. One minute is typically sufficient but use "top" to monitor for activity.

Jobee

My upgrade to R1 seem to go fine. The system seemed to rebooted and when I got back in it had 5 pkg to update. I tried the first one from GUI but it got stuck with the Please wait while the update system initializes. I went into SSH and did 13 to upgrade and it came back all upgraded. I did a complete reboot. This time I heard the chime of it shutting down and booting up. When it came up, I did not go to the GUI, but went back into SSH and went to the shell and did the grep pkg-install just to verify nothing was running, and it was not. Then did pkg update and it updated a number of packages. Then did pkg upgrade. Came back with the 5 I had to upgrade and let it fly. For the 5 pkgs it is going on 30 minutes and has just completed 2 of the 5. Normally updates are 2 minutes tops, but not sure what is taking the time this time around. My seems to take a long time at the Writing configuration.

Visseroth

@revengineer Thank you sir! That seemed to do the trick! I still have much to learn about command line but I am getting much more comfortable with it now that I've been using linux for a few years.

@Jobee I'm glad it's not just me, I mean I'm not glad you're having problems too but sometimes I think I'm the only one struggling on here at times. Try Revengineer's solution, it seemed to get the packages un-stuck and now the updates are running.

Edit: @Jobee You are right though, one package stopped at writing configuration and has been sitting there a while
What's your drive configuration?
I'm running 2 drives in a Raidz mirror configuration on this device, system is a SuperMicro C2758 w/16GB of RAM.

Jobee

I have 2 drives in mirror but only 4 GB ram

Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
2 CPUs: 1 package(s) x 2 core(s)

Been looking for an excuse to upgrade this old box. It always handles what I throw at it but 90% of the time it is running at 98% of memory and 60% of CPU.

Visseroth

@Jobee oh, yea, that's a older CPU.
I spent the money on the SuperMicro C2758 because it's 1U, 20W max, 8 core Intel Atom with ECC support and at the time Netgate was selling something similar on their website. It's rock solid and fast and idles at about 5 to 10% utilization with NTop, Snort, Squid, SquidGuard and PfBlocker all running. I'd be half tempted to use this machine as a small PC for basic stuff in a cabinet or my Ham radio stuff but at $800 just for the chassis and MB w/CPU it isn't cheap but cheaper than some PCs.

Well in other news I am having the same problem, writing configurations are VERY slow!!!!!!!

Jobee

@Visseroth I finally got mine to complete. I did use part of what @revengineer suggested. I did not have any pkg-static jobs but had just a pkg job stuck. I found it when I tried to run pkg upgrade again. It give me the same error you had at the beginning about being locked. I killed the job and then did update one more time. This time around I decided to run reach upgrade one by run. This helped. My pfBlocker took the longest at just over an hour. And just for good measure, I rebooted one more time at the end and then took a copy of the backup down to NAS.

I think I might start updating from the cli for all pkg and system updates.

Gertjan

Most probably "right now" most system get updated.
Somewhere, in a dark corner, some "pfSense package update server" gets hit really hard. With the understandable consequences.

revengineer

@Gertjan said in 2.4.5R1 Update no package re-install:

Most probably "right now" most system get updated.
Somewhere, in a dark corner, some "pfSense package update server" gets hit really hard. With the understandable consequences.

Would be nice for someone from Netgate to confirm this conjecture because aborting the package installation with "killall" did not leave me with a warm and fuzzy feeling.

EDIT: Note that I seemed to have had no issue downloading the packages, and they seemed to have installed fine, creating the GUI entries and working. It is just that the pkg-static process hung somewhere near the end. This does not point to an overloaded package server.

Visseroth

Ultimately I still couldn't get packages to finish and just restarted again to restart all processes which got pkg upgrade working again, it now confirms 4 packages needing updates and is in the process of doing the updates.

I started the reinstall of the pkgs yesterday at about 5pm it was still hung this morning at 9am.

Visseroth

Well seems to be still stuck............

Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (4 candidates): 100%
Processing candidates (4 candidates): 100%
Checking integrity... done (0 conflicting)
The following 4 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	pfSense-pkg-pfBlockerNG-devel: 2.2.5_32 -> 2.2.5_33 [pfSense]
	pfSense-pkg-snort: 3.2.9.11 -> 3.2.9.12 [pfSense]
	pfSense-pkg-squid: 0.4.44_25 -> 0.4.44_26 [pfSense]
	pfSense-pkg-squidGuard: 1.16.18_5 -> 1.16.18_6 [pfSense]

Number of packages to be upgraded: 4

Proceed with this action? [y/N]: y
[1/4] Upgrading pfSense-pkg-squidGuard from 1.16.18_5 to 1.16.18_6...
[1/4] Extracting pfSense-pkg-squidGuard-1.16.18_6: 100%
Removing squidGuard components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.

Visseroth

3 hours later this is still stuck.
Anymore suggestions?

Visseroth

7 hours later it was still in the exact same spot so I "killall pkg" and then ran pkg update then pkg upgrade and it is stuck again!!!
Seems 2.4.5R1 has some problems.

Any help appreciated, I'm not sure where to go from here.

Visseroth

Seems someone else is having the same problem on this thread....
https://forum.netgate.com/topic/154410/can-t-install-or-remove-packages/9

Followed this ... https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html
from that thread, didn't help, still stuck on "Writing configuration... done." and that is where it sits, no further movement.

just checked the logs and found this...
pid 98090 (ntopng), jid 0, uid 0: exited on signal 11 (core dumped)

Gertjan

@Visseroth said in 2.4.5R1 Update no package re-install:

ntopng

That would be a good candidate to put on a break while updating/upgrading.

Also, take a look at the newest Netgate's viddeo - the pfSEnse related one about fsck (see their Youtube channel).

Enough disk space left ? Memory ? Run top or htop in a different shell/ssh windows while upgrading : see what process are running 'hot' and if the systems starts to swap.

All this out of the top of my head.

Btw ; the pkg subsystem didn't change during upgrade.

edit :
More : as seen recently in another thread : IPv6 was used as it is the native protocol, before IPv4. If the IPv6 route between you, your ISP , and Netgate servers is bad, loading stuff from them start to be complicated.
I didn't even know such issues could happen.

edit : here https://forum.netgate.com/topic/154389/2-4-5r1-update-no-package-re-install/15

What also often eliminates a lot :
Install a clean pfSEnse from USB.
Then, assign your LAN & WAN, not more. Use default LAN settings, just do ONE thing know : make WAN work.
Period.
DO not use your own backed up config.xml !!!

Get the packages listed.
Install them by hand one by one .... no need to activate them. Check and verify after each step.
When your system is fully loaded, finish config by hand - use the config file as a guide (it's text editor reabable) or ... why not, import your config.xml.
Re test.

Be motivated and realize that ten of thousands of systems have been upgraded now .... so I tend to say the 'pkg' subsystem works. You do know that you and I and other use the same / identical code. Just our local system settings are different.
It's all about looking for that difference ... on your side.

sisko212

@revengineer i have same issue, and not only with as upgrade but also for a fresh 2.4.5p1 install followed by a configuration restore.
I always find pkg-install stuck and i have to kill the process manually after some minutes when i see it taking 0% cpu, in order to leave the upgrade script to continue with other packages... it is really a pain.

Gertjan

As said above :
Clean install.
[ The work-around start here ]
Install one by one the packages.
Only then import your config.xml.

See also https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available/52 and below.

Visseroth

I posted this message here... https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available/60

2.4.5 had issues, 2.4.5p1 has more problems, not just on my personal firewall but on another I support, system specs are exact.
I know the PfSense team does their best to ensure stability and reliability but I can't help but feel like the ball has been dropped somewhere.
In all my years of using PfSense I've never seen so many problems unless using a software NIC like realtek and when a user has a problem the answer should NEVER be "wipe and reload". For a commercial client this means down time, lots of hours trying to fix the problem or calling the PfSense technical support and having them fix the problem and now that I think about it, from a commercial standpoint is a great opportunity to make some extra cash whether intentional or not.

I was told with 2.4.5 that one should uninstall packages before updating.
Knowing user are running packages is pretty much a given so why should things break just because a update is being applied? Sure there's a lot of code to ensure everything goes smooth but this is a known variable in which updates are being applied.
2.4.5p1 the package installer breaks.... sometimes and I'm told the solutions is to reload from scratch, install the packages then load my config?

How is that an acceptable practice? I'm seriously afraid to upgrade any other systems I support because there is a likelihood that they will break and then what? I have to reinstall, preinstall the packages and then re-load the config?

I'm seriously thinking it may be time to leave PfSense and go to something like OPNSense or just forget a open based firewall and going to Ubiquiti.

Oh, and for all those haters that want to flame me, you're another reason I'm thinking of leaving.
I've basicaly said I have (and others) have a problem with the pkg installer and there has been no resolution.

Gertjan

@Visseroth said in 2.4.5R1 Update no package re-install:

2.4.5 had issues, 2.4.5p1 has more problems,

Read about it. That's all I experienced.
I'm using pfSense native and with VM's. No issue 'for me'.

@Visseroth said in 2.4.5R1 Update no package re-install:

personal firewall but on another I support, system specs are exact.

So, software is identical - hardware also.
You and I use the same software. Not the same hardware.
If you would be using my hardware - then no issues ?
(I'm just thinking out loud here)

@Visseroth said in 2.4.5R1 Update no package re-install:

should NEVER be "wipe and reload"

I agree.
Still, if issues, to hunt them down, it's just a way to discover what the issue is.
When I get my hands on a machine, I could do the testing without wiping everything.
Here, on the forum, we never know what happens on a device that shows issue.
Wiping is advised because it makes sure the software platform is ok - and their could only be "settings" issues.

Upgrading pfSense Software

[Netgate Blog : pfSense 2.4.5-RELEASE-p1 Now Available](Upgrading pfSense Software)

=>

Do not update packages before upgrading pfSense! Either remove all packages or do not update packages before running the upgrade.

I just hit the upgrade button - that's it.
I use

acme 	0.6.8 	
Avahi 	2.1_1 	
Cron 	0.3.7_4 	
freeradius3 	0.15.7_16 	
Notes 	0.2.9_2 	
nut 	2.7.4_7 	
openvpn-client-export 	1.4.23 	
pfBlockerNG-devel 	2.2.5_33 	
RRD_Summary 	2.0 	
Shellcmd 	1.0.5_1 	
System_Patches

some of these were upgraded after 2.4.5-P1 came out.

@Visseroth said in 2.4.5R1 Update no package re-install:

and there has been no resolution ...

There is always a solution, often minutes after an issue is recognized and/or can be reproduced at will.
It boils down to : the one that "codes things up" should be aware of the issue. After that, things will go fast.

@Visseroth said in 2.4.5R1 Update no package re-install:

there is a likelihood that they will break and then what?

Yep. That can happen. Even Microsoft, with a WIN10 update, managed to brake the DHCP client while updating, leaving the "PC" without a connection afterwards.

Normally, pfSense can be re installed very fast : keep the N-1 version a USB key (if you have it, you will never need it), and back up your config often (I have it exported every day at 16h00 - and there is the auto backup form Netgate).

I'll do a console reset this week with my @home system.
Or even re install it 'from scratch'. As soon as interface are assigned, I'll import my "config.xml" and have packages being installed during that process. I'll post back with the results.

follysuperscript

Just chiming in to add that I'm having the same issue.

My upgrade from 2.4.5 went fine, but after a recent hard drive replacement, reinstalling and restoring by config is hanging at package re-install like others in this thread. I've tried 3 times with the same results.

Running on bare metal.

Though killing the hung processes is the only thing that got me back up and running, I'm noticing some strange boot up delays at "starting package xxxx ....done", everything appears to be working so it might be nothing, or it might be unrelated, but I worry about killing a process during install.

To all involved in this this great software, thanks! I've used it for many years now without issue.