After Update 2.7.2 / 23.09.1

jrey

Fair enough - pretty sure back in 22.?? the pfb update didn't appear until checking the packages pages and had to be updated from the packages page manually. (maybe that is just because the package wasn't actually released until a short time after the system update in that case)

_7 was released well before 23.09.1 release, but had not been installed on this system.

Thanks for the clarity

Though I would not expect that to show in the system uildtime like that.

the _7 release was built and released well before the 9th. it was installed and running on our test machine before it was upgraded Friday to 2.7.2
So what we are seeing/suggesting is a released package is apparently being built with the same version number after release and slipstreamed into the full release. To us that is just bad practice.
if someone updates to 23.09.1 a week from now, are they likely to get yet a different build date? (what if the build has a flaw, for any reason, how do we tell them apart they are all _7)
There might be a perfectly good reason for rebuilding a released package, but generally that should/would go hand in hand with a new rev. in the version.

stephenw10

There is only ever one version of the package in the repo. When you upgrade pfSense versions it upgrades any pkgs available. So after an upgrade you should always have all the current pkg versions. That has always been the case.

None of that explains the unexpected build timestamp though.

Do you see the output of:
cat /etc/version.buildtime

match:
pkg-static info -A pfSense | /usr/bin/awk '/build_timestamp/ {print $2;}'

?

jrey

@stephenw10

Interesting,

so for the first one I already post it above (but still is)
20231209-1757

for the pkg-static it is
2023-12-09T19:39:03+0000

and from the boot log it is
FreeBSD 14.0-CURRENT aarch64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 20:59:18 UTC 2023

The 9th being Sat, the official release before that and the install being Sunday the 10th.

the 2.7.2 test box as mentioned, the dates are more in line. It was installed Friday (8th) and all three dates there are on the 6th.

sounds like the repo perhaps got something different than what was first released sometime between Sat and Sun morning ?

the test box is a virtual, I wonder what would happen if I went back to the snapshot at 2.7.1 and applied the update again now. That might be a test later, if time permits. Other fish in the pond right now.
(Although I could, I am not going to do this on the production box, by going back to the ZFS snap. Seems working fine. Don't fix it.)

This certainly isn't a crisis by the way. Just seems odd.

stephenw10

Yup it's just confusing this close to a release. In fact I believe it's expected. Still digging though..

jrey

@stephenw10

Don't dig to hard...

is just odd that once a product is released the repo would end up with newer builds. A test repo sure, but production just seems like it should be frozen in time until a new release requires a new repo.

Maybe the "new" build was just to accommodate whatever took the repo's offline for most of yesterday. (although they went away shortly after I did the upgrade, and then remained unavailable for several hours)

jrey

@stephenw10

Don't want to keep hijacking the other thread or create a new one for this really. So this will be completely out of context on this thread and people can just scratch their head and ignore it

however over there you said:

Open a feature request for it if you want. However that is the expected behaviour it's not a bug.

Sorry, but anything that allows Joe User to shoot themselves in the foot (and this does) should not be "expected behaviour".

It is the current behaviour, true, i recognized that right away, but it should not be labeled as "expected" just because it is "current"

Your comment as a Netgate rep, makes me not sure I really care enough about it to create a change request either. It's not really a problem I have. I recognized what it was/is doing right away and know what it's doing. But Joe U might not, but the position it is "expected" is typical of so many companies. When they know full well it has the potential to break stuff for Joe U and elect to call it expected not a bug. Just baffles me.

-my 2 cents

I've removed my messages on the subject from the other thread.

stephenw10

If you add a feature request then other devs can make comments or suggestions on there.

My own opinion is that if you are able to add custom patches then we are limited in what we can or should do to prevent foot shooting. There's nothing to stop someone adding a custom patch that breaks everything whether that be applying or reverting. We could prevent adding custom patches that way at all but there are a lot of people using that. We could perhaps improve the warnings around using the package at all.

jrey

@stephenw10 said in After Update 2.7.2:

limited in what we can or should do

respectfully disagree. You can as I suggested:

display a warning on save (of new one that can't actually apply)
and
during / after an upgrade scan the patches and do the same determination, don't just automatically set the ones that are auto apply to apply if the changes already exist. flag them in the GUI as not required.

If I find some cycles, I'd actually consider "patching" this myself. That's how serious the problem could be for our friend Joe U.

There's nothing to stop someone adding a custom patch that breaks everything

That's true, you can't fix stu..pid. But that's not the case here - this is 100% a preventable situation on both counts, and the system should therefore protect itself and remain flexible as much as it can within the framework.

lot of people using that

Yup for sure. Me included. That's the main way I fix stuff that is "broken" that I know will never get fixed, but is required for operation in our environment -- I have a laundry list (actually a set of 8 patches at this time) that get reverted before every major update - re-evaluated with every new release and reapplied as required. ... I'm not open to discussion what those 8 items are in an open forum.

But if in some cases the redmine is closed as "not required" there is generally no point in continuing - just easier to fix it and move on.

Over and above those 8, there are redmine items (for example one we (you and I as I recall, but could be wrong on that) have discussed in the past (regarding the Traffic Graph Widget of all things)
That redmine is showing as affected version 23.05.1 Status New and 0% done, even though there is a patch on the ticket not a single comment since 10/28/2023 except one person who commented (a month ago) that "I can reproduce this in version 23.09"
I guess I could also just go on to redmine and comment saying "still in 23.09.1, but the patch still works." Seems futile since it hasn't even been looked at, in 2 months. Bigger fish and all.

Thanks for the banter,

EyeTap

What was seconds, is now in the minutes category.

I can confirm this observation - interestingly not right after the upgrade though but after 24 hrs after a final reboot.

Here fetching both the installed and available packages takes like 30 seconds.
The Update page takes about 30 seconds to open and further 30 seconds to check if the installation is up to date or not.
Opening the update settings page then also takes ~30 seconds to load...

I came from 2.7.1 where I couldn’t notice this behaviour - pfsense is running on a physical box, CPU usage is low and the bandwidth is fine, so I can’t see any bottleneck there…

More an annoyance than a true issue though….

jrey

@EyeTap

Here fetching both the installed and available packages takes like 30 seconds.

not here those pages are pretty quick maybe around 10-15 seconds to open and display the packages.

The Update page takes about 30 seconds to open

longer here, really haven't looked into it. this is where the delay is for me, just loading the page. and switching the tab is just as bad, that doesn't make sense.

further 30 seconds to check if the installation is up to date or not.

once it loads, another 10-20 seconds is typical for checking if there is something available. This would be while the cog is spinning (this seems pretty normal)

It's not really spending a lot of time going to the page. So almost never. There is enough noise when a new version is available, it becomes really obvious when it is time to check it.

That said, still the same observation today.
so for me this item is filed under category "of it is what it is"

stephenw10

@jrey said in After Update 2.7.2:

That's true, you can't fix stu..pid. But that's not the case here - this is 100% a preventable situation on both counts, and the system should therefore protect itself and remain flexible as much as it can within the framework.

As I say open a feature request for it. You may find other developers agree with you. I wouldn't be me coding that anyway.

jrey

@stephenw10 said in After Update 2.7.2:

Still digging though..

Add this to the low priority "dig" re packages (dig being the reference for unbound )

in the update ( 23.09.1)
the update log shows

unbound: 1.18.0 -> 1.18.0_1 [pfSense]

Number of packages to be upgraded: 9
[1/9] Upgrading unbound from 1.18.0 to 1.18.0_1...
===> Creating groups.
Using existing group 'unbound'.
===> Creating users
Using existing user 'unbound'.
[1/9] Extracting unbound-1.18.0_1: .......... done

unbound is working no issue, but it is logging itself as 1.18.0
and running the command

unbound -V

reports 1.18.0

where as in the 2.7.2 release there is no mention of unbound being upgraded to _1

unbound -V also reports 1.18.0 on that version
maybe that's the package the got slipped in ? or is expected

either way on 23.09.1 it is not reporting what is says was installed..

We can talk about the curl package too.. Upgraded on 23.09.1 now running 8.5.0 but 2.7.2 is still running 8.4.0 maybe that is also expected?

from 23.09.1 upgrade

Installed packages to be UPGRADED:
	curl: 8.4.0 -> 8.5.0 [pfSense]
	isc-dhcp44-relay: 4.4.3P1_3 -> 4.4.3P1_4 [pfSense]
	isc-dhcp44-server: 4.4.3P1_3 -> 4.4.3P1_4 [pfSense]
	openvpn: 2.6.5 -> 2.6.8_1 [pfSense]
	pfSense: 23.09 -> 23.09.1 [pfSense]
	pfSense-default-config-serial: 23.09 -> 23.09.1 [pfSense]
	pfSense-repo: 23.09 -> 23.09.1 [pfSense]
	strongswan: 5.9.11_1 -> 5.9.11_3 [pfSense]
	unbound: 1.18.0 -> 1.18.0_1 [pfSense]

from 2.7.2 upgrade

Installed packages to be UPGRADED:
	isc-dhcp44-relay: 4.4.3P1_3 -> 4.4.3P1_4 [pfSense]
	isc-dhcp44-server: 4.4.3P1_3 -> 4.4.3P1_4 [pfSense]
	openvpn: 2.6.7 -> 2.6.8_1 [pfSense]
	pfSense: 2.7.1 -> 2.7.2 [pfSense]
	pfSense-default-config: 2.7.1 -> 2.7.2 [pfSense]
	pfSense-repo: 2.7.1 -> 2.7.2 [pfSense]
	strongswan: 5.9.11_2 -> 5.9.11_3 [pfSense]

stephenw10

The curl pkg was updated in the repo. Upgrades after it was will have 8.5.0. Anything that was already upgraded can get it using pkg upgrade if needed.

The Unbound package had the port revision bumped to pull in a fix to a patch we carry:
https://github.com/pfsense/FreeBSD-ports/commit/4749e98e2dbffb8e725f68e5b437c5056c55a0bc

jrey

@stephenw10

Thanks

so the question about unbound wasn't so much about the difference between 2.x and 23.x but more about 23.x saying it is still reporting itself as 1.18.0 when it should be 1.18.0_1 ? is that the expectation?

unbound -V
reports 1.18.0

sounds like these are the parts that caused the Dec 9th date on the install completed on Sunday morning. Seems to answer that. Packages were added a couple of days after the release, and earlier installers will be "missing" or running older versions of those packages. I guess what they don't know won't hurt them.

Thanks for the clarity.

Patch

@jrey said in After Update 2.7.2 / 23.09.1:

So what I think you are saying that packages on a released build are changing in the background and after the release?
therefore those folks that updated earlier will have a different system than those who update later.. seems like the potential for a support nightmare.

@stephenw10 said in After Update 2.7.2 / 23.09.1:

We commonly upgrade packages for fixes between releases when required.

@stephenw10 said in After Update 2.7.2 / 23.09.1:

Anything that was already upgraded can get it using pkg upgrade if needed.

The Unbound package had the port revision bumped to pull in a fix to a patch we carry:
https://github.com/pfsense/FreeBSD-ports/commit/4749e98e2dbffb8e725f68e5b437c5056c55a0bc

Err
How is a user expected to know

a package such as unbound has been updated in the Netgate repro and
if they had delayed their update they would have got it but
on their install they have not as they updated early.

or does this mean users should intermittently run pkg upgrade just to in case as the "Upto to date" status may not be accurately reported in System -> Update -> System Update

jrey

@Patch

My Opinion... no you shouldn't have to --

when they "release" a system the repo for that Release version should be frozen at that version. You should never get a different bundle on Monday compared to what you got on Friday, or any other date span.

If they have to add something that was missed or late to the party. it should require a new repo.. ie 23.09.2 (or 23.09.1p1)

Slipstreaming packages into a released product is just bad practice for oh so many reasons.

if you look at my comments on this thread, even the announcement of the release wasn't "clean"

https://forum.netgate.com/topic/184642/netgate-releases-pfsense-plus-software-version-23-09-1-and-pfsense-ce-v2-7-2/7?_=1702381351314

Currently I'm trying to get through a security audit (these happen periodically in the real world) There has to be documentation on what changes are applied, when, by whom etc. Versions numbers typically have to match or auditors tends to get cranky. (so for example the unbound says 1.18.0_1 was installed but the package itself reports 1.18.0 this is NOT a pass, guaranteed. -
They are not running with the "big boys" in regards to releases and control.

I mean I get it, their lower end line of products are not intended for secure environments. But still basic adherence to release practices should not be that hard.

stephenw10

@jrey said in After Update 2.7.2 / 23.09.1:

more about 23.x saying it is still reporting itself as 1.18.0 when it should be 1.18.0_1 ? is that the expectation?

That is expected. Unbound itself, the compiled binary, is still 1.18.0 and so will report that when you call it. The port version was bumped to _1, you should see that from pkg info unbound

jrey

@stephenw10

yes I know, but that's not necessarily what the audit looks for.
They look for the what was installed see 1.18.0_1 and what is running "unbound -V" and logs both say 1.18.0

then it turns into a big long discussion about why, to mitigate the concern.

If the versioning just matched there would be no question. period.

johnpoz

@jrey said in After Update 2.7.2 / 23.09.1:

If the versioning just matched there would be no question. period.

I don't think that is going to happen with something like this - freebsd backported some stuff from 1.19 to the 1.18 release I believe.. There is not actual 1.18.0_1 from unbound (nlnetlabs).. see here

https://www.freshports.org/dns/unbound/?page=1#history

Or maybe the came up with their own fix and provided it in their package.

This is not uncommon in packages for linux and or bsd distros..

stephenw10

I understand. I'm not sure what we can do about it though. That's how the ports system has always worked AFAIK.