To do 25.07 or not?! That is the question!

JD 0

Just did the upgrade from 25.07RC to 25.07RELEASE.; Absolutely no issues whatsoever. On a 4200 the total downtime was < 2 mins. I say that because monitoring didn't trigger except to log the reboot itself.

Environment:
Netgate 4200
LAGG interfaces
Multiple VLANS with an IOT secure cell
Dual stack IPv4/v6 (RA managed mode with DHCPv6/DDNS)

Zermus

I'd wait. I just bricked a VM doing it from 24.11. Got 403 Forbidden soon as I pushed the upgrade button and SSH stopped working, or more accurately it would auth then immediately boot me out... and it just locked everything up for 30 minutes. I finally tried a reboot and the rc.init was all screwed up. I just reloaded from the last snapshot.

Now I have another mini PC with the same issue, no Proxmox snapshot I can backup on that one.... ugh.... I may have to rebuild that one from scratch... son of a.....:(

provels

@Zermus Same thing here on physical. Oh, well.

Gertjan

@chudak

Same thing here : SG-4100 and it really looks like 25.07 was build for it
Its up and running now for two days, all is well, nothing goes of the charts.
Captive portal clients can still connect (its high season) - no one is yelling here.

@mcury said in To do 25.07 or not?! That is the question!:

Service_Watchdog

Really ? The best known system killer out there.

It has been ages for me that processes died on me.

edit : ah, ok, Rebel Alliance - I get it ;)

mcury

@Gertjan said in To do 25.07 or not?! That is the question!:

Same thing here : SG-4100 and it really looks like 25.07 was build for it

Indeed =)

It has been ages for me that processes died on me.

In version 24.11, I had issues with the NUT service failing to start on boot. That's why the service watchdog, only for NUT.
I’ll try disabling it in this new version 25.07 to see how that goes now.

edit : ah, ok, Rebel Alliance - I get it ;)

ahahahah, nothing like fresh adrenaline in the morning

Edit:

One thing I noticed is the disk IO decreased?
iostat -x is showing now around 50, it was around 75 before, can you confirm ?

Gertjan

@mcury said in To do 25.07 or not?! That is the question!:

In version 24.11, I had issues with the NUT service failing to start on boot. That's why the service watchdog, only for NUT.
I’ll try disabling it in this new version 25.07 to see how that goes now.

Wait .....
Thanks !!
I still see that : after a upgrade-reboot and normal reboot (?) the UPS service is shown down on the dashboard. After hitting 'Save' on the Services > UPS > Settings page, it's up and running. Never actually took some time to investigate why it doesn't start on boot.
So, I could (ab)use the "service watchdog" for this .... interesting
Thanks again for the suggestion.

JD 0

I saw similar behavior in NUT several releases back (don't recall which). But it's not resurfaced since. The instance running is polling the UPS over SNMP.

Zermus

FWIW doing a "pfSense-upgrade -d" from CLI fixes this for me and does the upgrade properly. Not sure why that works and the GUI fails lol. I did have to rebuild my base packages. Here is what ChatGPT had to say about it. I had the same problem, two different locations, network providers, etc. One is in a datacenter with multiple network redundancies so I doubt it was a network issue.

Root cause:
The core problem was due to an incomplete or partially failed upgrade from pfSense 24.11 to 25.07. The missing critical libraries (libmd.so.7), corrupted package repositories, and broken package signatures indicate that some part of the upgrade script was interrupted, incomplete, or encountered dependency conflicts.
Specific indicators of broken upgrade:
Missing libraries (libmd.so.7) causing package operations to fail.

Missing critical files (/usr/local/sbin/read_global_var, /usr/local/libexec/pfSense-upgrade, and /etc/version) indicate that pfSense-base or core packages were only partially upgraded.

Invalid or broken repository signatures (pkg-static: Error loading trusted certificates) point to repository configuration or trust issues post-upgrade.

Dependency conflicts (IGNORE_OSVERSION prompts) clearly indicated version mismatches due to packages from different pfSense/FreeBSD versions.

AndyRH

Updated my 7100. Took less than 5 minutes.

For some unknown reason the CPU usage has dropped. Even with a speed test pushing 1.29Gb the CPU only gets to about 50%. Yesterday that same test was pushing 80%. Crazy...

stephenw10

Hmm, what version were you seeing 80% usage in?

AndyRH

@stephenw10 24.11 would idle above 30% and speed test would push it over 80%. The only change was the upgrade.
I never saw a performance problem so I never chased it.
For a short time I had 2Gb internet and it would hit 100% pushing 2.5Gb which was the limit of the test machine.

stephenw10

Hmm, interesting! PPPoE WAN with the new driver maybe?

AndyRH

@stephenw10 Fiber straight into the 7100. Nothing funky. No heavy packages.
With no performance hit I did not see a reason to chase it.

Someone fixed something and made it better.

In theory I could boot the old image and Clonezilla the disk.

stephenw10

@AndyRH said in To do 25.07 or not?! That is the question!:

Someone fixed something and made it better.

I'll take it.

chudak

Finally upgraded 24.03 to 25.07.1

After the initial reboot the network seemed to be messed up, I could not even ping or login to my router.

However after the power off and on everything looks normal.

I did skip the previous major update as my system was rebooting by itself every 5-11 minutes.

So now need to see that that problem is gone.

Cross my fingers :)

List of installed packadges:

UPDATE

The problem is not fixed :(

I see every ~17 minutes system kinda reboots.

The reason I say "kinda" that last shows one reboot I did after upgrade

[25.07.1-RELEASE][admin@pfsense.wawona.lan]/root: last -4 reboot
shutdown time Tue Aug 19 13:23

But up-time in the UI goes up to ~17 min, then I see services go down and then back up again

And that's every ~17 min!!!

Anybody any clues?

stephenw10

What do you see logged when that happens?

What are you running on?

chudak

@stephenw10 said in To do 25.07 or not?! That is the question!:

What do you see logged when that happens?

What are you running on?

QOTOM-Q355G4

UPDATE:
https://www.youtube.com/watch?v=2RR26J-dDzc

Spent 2 h trying to see what could have caused reboots/resets/up/down unsuccessfully and decided to restore the 24.03 boot env. And ... it did not work, verification during the boot failed!!!! And then another one did not work either etc. Finally I got old 23.x version booted. But I realized that I need to do a fresh install.

Thx to AI for helping to figure what key to press for BIOS setup :)

Long story short, I did fresh install + restore old config.xml and so been running for 24 + min (before it was rebooting every 17)

So hope I am out of the woods....

stephenw10

So a clean install of 25.07.1 is running OK?

Hmm, hard to imagine would could have caused that.

chudak

@stephenw10 said in To do 25.07 or not?! That is the question!:

So a clean install of 25.07.1 is running OK?

Hmm, hard to imagine would could have caused that.

Well there are several issues

The s/w change from 24.03 + caused issues. Support's assumption it was h/w issue turned out to be untrue
This is a difficult problem to find a root cause for and my route h/w may have played a role as well.
Boot environments restore failed for multiple snapshots. That is really bad. I have no system around anymore for troubleshooting but I hope somebody can reflect and think about what could have caused that. That was scary, the whole reason I did not to want fresh reinstall because my false feeling that boot envs were safe. Alas :(

Antibiotic

@chudak Looks like on 25.07.1 AES-GSM broken, have a speed 2oo mb with openVPN client and DCO enabled. Before on 24.11 has a speed almost 1 gb with openVPN. https://forum.netgate.com/topic/198535/massive-10x-performance-regression-in-aes-gcm/7