2.4.5R1 Update no package re-install

revengineer

@Gertjan said in 2.4.5R1 Update no package re-install:

Most probably "right now" most system get updated.
Somewhere, in a dark corner, some "pfSense package update server" gets hit really hard. With the understandable consequences.

Would be nice for someone from Netgate to confirm this conjecture because aborting the package installation with "killall" did not leave me with a warm and fuzzy feeling.

EDIT: Note that I seemed to have had no issue downloading the packages, and they seemed to have installed fine, creating the GUI entries and working. It is just that the pkg-static process hung somewhere near the end. This does not point to an overloaded package server.

Visseroth

Ultimately I still couldn't get packages to finish and just restarted again to restart all processes which got pkg upgrade working again, it now confirms 4 packages needing updates and is in the process of doing the updates.

I started the reinstall of the pkgs yesterday at about 5pm it was still hung this morning at 9am.

Visseroth

Well seems to be still stuck............

Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (4 candidates): 100%
Processing candidates (4 candidates): 100%
Checking integrity... done (0 conflicting)
The following 4 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	pfSense-pkg-pfBlockerNG-devel: 2.2.5_32 -> 2.2.5_33 [pfSense]
	pfSense-pkg-snort: 3.2.9.11 -> 3.2.9.12 [pfSense]
	pfSense-pkg-squid: 0.4.44_25 -> 0.4.44_26 [pfSense]
	pfSense-pkg-squidGuard: 1.16.18_5 -> 1.16.18_6 [pfSense]

Number of packages to be upgraded: 4

Proceed with this action? [y/N]: y
[1/4] Upgrading pfSense-pkg-squidGuard from 1.16.18_5 to 1.16.18_6...
[1/4] Extracting pfSense-pkg-squidGuard-1.16.18_6: 100%
Removing squidGuard components...
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.

Visseroth

3 hours later this is still stuck.
Anymore suggestions?

Visseroth

7 hours later it was still in the exact same spot so I "killall pkg" and then ran pkg update then pkg upgrade and it is stuck again!!!
Seems 2.4.5R1 has some problems.

Any help appreciated, I'm not sure where to go from here.

Visseroth

Seems someone else is having the same problem on this thread....
https://forum.netgate.com/topic/154410/can-t-install-or-remove-packages/9

Followed this ... https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html
from that thread, didn't help, still stuck on "Writing configuration... done." and that is where it sits, no further movement.

just checked the logs and found this...
pid 98090 (ntopng), jid 0, uid 0: exited on signal 11 (core dumped)

Gertjan

@Visseroth said in 2.4.5R1 Update no package re-install:

ntopng

That would be a good candidate to put on a break while updating/upgrading.

Also, take a look at the newest Netgate's viddeo - the pfSEnse related one about fsck (see their Youtube channel).

Enough disk space left ? Memory ? Run top or htop in a different shell/ssh windows while upgrading : see what process are running 'hot' and if the systems starts to swap.

All this out of the top of my head.

Btw ; the pkg subsystem didn't change during upgrade.

edit :
More : as seen recently in another thread : IPv6 was used as it is the native protocol, before IPv4. If the IPv6 route between you, your ISP , and Netgate servers is bad, loading stuff from them start to be complicated.
I didn't even know such issues could happen.

edit : here https://forum.netgate.com/topic/154389/2-4-5r1-update-no-package-re-install/15

What also often eliminates a lot :
Install a clean pfSEnse from USB.
Then, assign your LAN & WAN, not more. Use default LAN settings, just do ONE thing know : make WAN work.
Period.
DO not use your own backed up config.xml !!!

Get the packages listed.
Install them by hand one by one .... no need to activate them. Check and verify after each step.
When your system is fully loaded, finish config by hand - use the config file as a guide (it's text editor reabable) or ... why not, import your config.xml.
Re test.

Be motivated and realize that ten of thousands of systems have been upgraded now .... so I tend to say the 'pkg' subsystem works. You do know that you and I and other use the same / identical code. Just our local system settings are different.
It's all about looking for that difference ... on your side.

sisko212

@revengineer i have same issue, and not only with as upgrade but also for a fresh 2.4.5p1 install followed by a configuration restore.
I always find pkg-install stuck and i have to kill the process manually after some minutes when i see it taking 0% cpu, in order to leave the upgrade script to continue with other packages... it is really a pain.

Gertjan

As said above :
Clean install.
[ The work-around start here ]
Install one by one the packages.
Only then import your config.xml.

See also https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available/52 and below.

Visseroth

I posted this message here... https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available/60

2.4.5 had issues, 2.4.5p1 has more problems, not just on my personal firewall but on another I support, system specs are exact.
I know the PfSense team does their best to ensure stability and reliability but I can't help but feel like the ball has been dropped somewhere.
In all my years of using PfSense I've never seen so many problems unless using a software NIC like realtek and when a user has a problem the answer should NEVER be "wipe and reload". For a commercial client this means down time, lots of hours trying to fix the problem or calling the PfSense technical support and having them fix the problem and now that I think about it, from a commercial standpoint is a great opportunity to make some extra cash whether intentional or not.

I was told with 2.4.5 that one should uninstall packages before updating.
Knowing user are running packages is pretty much a given so why should things break just because a update is being applied? Sure there's a lot of code to ensure everything goes smooth but this is a known variable in which updates are being applied.
2.4.5p1 the package installer breaks.... sometimes and I'm told the solutions is to reload from scratch, install the packages then load my config?

How is that an acceptable practice? I'm seriously afraid to upgrade any other systems I support because there is a likelihood that they will break and then what? I have to reinstall, preinstall the packages and then re-load the config?

I'm seriously thinking it may be time to leave PfSense and go to something like OPNSense or just forget a open based firewall and going to Ubiquiti.

Oh, and for all those haters that want to flame me, you're another reason I'm thinking of leaving.
I've basicaly said I have (and others) have a problem with the pkg installer and there has been no resolution.

Gertjan

@Visseroth said in 2.4.5R1 Update no package re-install:

2.4.5 had issues, 2.4.5p1 has more problems,

Read about it. That's all I experienced.
I'm using pfSense native and with VM's. No issue 'for me'.

@Visseroth said in 2.4.5R1 Update no package re-install:

personal firewall but on another I support, system specs are exact.

So, software is identical - hardware also.
You and I use the same software. Not the same hardware.
If you would be using my hardware - then no issues ?
(I'm just thinking out loud here)

@Visseroth said in 2.4.5R1 Update no package re-install:

should NEVER be "wipe and reload"

I agree.
Still, if issues, to hunt them down, it's just a way to discover what the issue is.
When I get my hands on a machine, I could do the testing without wiping everything.
Here, on the forum, we never know what happens on a device that shows issue.
Wiping is advised because it makes sure the software platform is ok - and their could only be "settings" issues.

Upgrading pfSense Software

[Netgate Blog : pfSense 2.4.5-RELEASE-p1 Now Available](Upgrading pfSense Software)

=>

Do not update packages before upgrading pfSense! Either remove all packages or do not update packages before running the upgrade.

I just hit the upgrade button - that's it.
I use

acme 	0.6.8 	
Avahi 	2.1_1 	
Cron 	0.3.7_4 	
freeradius3 	0.15.7_16 	
Notes 	0.2.9_2 	
nut 	2.7.4_7 	
openvpn-client-export 	1.4.23 	
pfBlockerNG-devel 	2.2.5_33 	
RRD_Summary 	2.0 	
Shellcmd 	1.0.5_1 	
System_Patches

some of these were upgraded after 2.4.5-P1 came out.

@Visseroth said in 2.4.5R1 Update no package re-install:

and there has been no resolution ...

There is always a solution, often minutes after an issue is recognized and/or can be reproduced at will.
It boils down to : the one that "codes things up" should be aware of the issue. After that, things will go fast.

@Visseroth said in 2.4.5R1 Update no package re-install:

there is a likelihood that they will break and then what?

Yep. That can happen. Even Microsoft, with a WIN10 update, managed to brake the DHCP client while updating, leaving the "PC" without a connection afterwards.

Normally, pfSense can be re installed very fast : keep the N-1 version a USB key (if you have it, you will never need it), and back up your config often (I have it exported every day at 16h00 - and there is the auto backup form Netgate).

I'll do a console reset this week with my @home system.
Or even re install it 'from scratch'. As soon as interface are assigned, I'll import my "config.xml" and have packages being installed during that process. I'll post back with the results.

follysuperscript

Just chiming in to add that I'm having the same issue.

My upgrade from 2.4.5 went fine, but after a recent hard drive replacement, reinstalling and restoring by config is hanging at package re-install like others in this thread. I've tried 3 times with the same results.

Running on bare metal.

Though killing the hung processes is the only thing that got me back up and running, I'm noticing some strange boot up delays at "starting package xxxx ....done", everything appears to be working so it might be nothing, or it might be unrelated, but I worry about killing a process during install.

To all involved in this this great software, thanks! I've used it for many years now without issue.

YonNomNom 0

I just had this issue and I set the option for the firewall to use IPv4 even if IPv6 was available and it seemed to make it work but it could be a fluke.

follysuperscript

@YonNomNom-0
For reference to others looking for a cause, my config is IPv4 only. I have IPv6 turned off / blocked in multiple locations.

Visseroth

For all those having the same problem here is what I did to get the pkgs to install...
I did this from command line but I assume it can be done via the web gui as well.

For me in command line I did...

killall pkg-static
pkg-static upgrade -f

When and if the pkg hangs go into the web gui and go to Status --> Services and stop the packages that you are hung on. You may need to stop the pkg multiple times because it seems the pkg automatically restarts quite quickly.
Be sure to give each package a minute or two before stopping it. I saw a couple packages take a minute or so before continuing.

Oh! and if you get the famous lock error I had to reboot to get the lock to release, no killing of any kind managed to get the lock to let loose.

And many thanks to @al @revengineer for helping me with this problem!!!

revengineer

@Visseroth said in 2.4.5R1 Update no package re-install:

When and if the pkg hangs go into the web gui and go to Status --> Services and stop the packages that you are hung on. You may need to stop the pkg multiple times because it seems the pkg automatically restarts quite quickly.

This is essentially the workaround that @jimp proposed here a few days ago. I am glad you got it to work.

Visseroth

@revengineer Ahh, never did see that thread. Well I'm glad it's a known issue and I hope it gets fixed. Thanks for pointing that out and thanks for your help