Upgrade files are a lot larger than expected

phil.davis · Oct 1, 2015, 8:43 AM

https://snapshots.pfsense.org/i386/pfSense_master/updates/?C=M;O=D
For example, 2GB i386 nanoBSD upgrade file is 138,216,898

The corresponding 2.2.5 upgrade files are around 85MB.

I will try it soon on a test system anyway, but is this size increase expected?
If so, what are the main contributing factors?

I do not use 1GB CF cards, but for people who do, will this go close to filling up a 1GB CF card (given that there are 2 slices, expansion of the compressed upgrade image…)?

phil.davis · Oct 1, 2015, 9:55 AM

Note: I successfully upgraded a test Alix system from 2.2.5-DEVELOPMENT to 2.3-ALPHA using pfSense-2.3-ALPHA-2g-i386-nanobsd-upgrade-20151001-0124.img.gz - so it works.

I guess that the new build is bigger because it is much more based on the standard FreeBSD distribution and we get a bunch more standard utilities built in by default.

rbgarga · Oct 1, 2015, 10:36 AM

@phil.davis:

Note: I successfully upgraded a test Alix system from 2.2.5-DEVELOPMENT to 2.3-ALPHA using pfSense-2.3-ALPHA-2g-i386-nanobsd-upgrade-20151001-0124.img.gz - so it works.

I guess that the new build is bigger because it is much more based on the standard FreeBSD distribution and we get a bunch more standard utilities built in by default.

FullUpdate images will be used only when you are upgrading from 2.2.x to 2.3. After that upgrade process is based on pkg. GUI is not finished yet, please use console menu option 13 to upgrade to newer snapshots.

About the image size, it increased because we are now distributing full packages instead of copy individual files and dependencies based on a copy list.

jimp · Oct 1, 2015, 11:22 AM

It's also bigger because we are keeping more things from the FreeBSD base in place, bringing us closer to parity with FreeBSD.

robi · Oct 1, 2015, 11:56 AM

Will we still have 1GB NanoBSD image available in 2.3?

phil.davis · Oct 1, 2015, 12:00 PM

@robi:

Will we still have 1GB NanoBSD image available in 2.3?

Image for 1GB card is built. If you have a 1GB card and a test system to try it on, why not install 2.3-ALPHA on it and report back how much disk space is used/free.

jimp · Oct 1, 2015, 12:05 PM

It's tough to say. It's close. May not make it. May be time for it to go the way of the 512MB card.

4GB+ cards have been cheap and ubiquitous for years now, not much reason to stay with such small cards or small images. If it's still actually a 1GB card in service, odds are it is worth replacing with fresh media.

robi · Oct 1, 2015, 2:08 PM

We have tens of devices in production with built-in 1GB DOMs. These are hard and expensive to replace.

We'd be even happy with a special lite version, containing only one OS slice.

jimp · Oct 1, 2015, 2:13 PM

At this point I wouldn't trust a 1GB DOM… how old are those?

You could just run a full install, though upgrading an exiting install for cases may be tricky/impossible.

I doubt we'll be making a "lite" version like that, though we may yet find things to trim away to bring the size down a little more, I doubt it will be a drastic difference.

robi · Oct 2, 2015, 11:59 AM

These are branded, rock-solid HP thin clients. Never failed because of the DOM.

Even a 98% filled up image would be fine, it won't allow installing packages, but still, it would work great as a standard pfSense box.

jimp · Oct 2, 2015, 12:05 PM

Then run a full install on them and not NanoBSD. Problem solved. :-)

Though you may still have trouble with updates on a disk that small.

That said, I don't care how "rock solid" something is, age will take it down eventually, and you should have a plan in place for when it does.

robi · Oct 5, 2015, 9:39 AM

@phil.davis:

@robi:

Will we still have 1GB NanoBSD image available in 2.3?

Image for 1GB card is built. If you have a 1GB card and a test system to try it on, why not install 2.3-ALPHA on it and report back how much disk space is used/free.

OK, I've installed the 1GB NanoBSD image, I'm satisfied with the fact that it only occupies 89% of the system partition, and it still contains a duplicate slice copy of it.
Maybe an exact brush up of the installed packages can recover a bit, but for a simple router that's perfectly fine.

robi · Oct 5, 2015, 9:40 AM

@phil.davis:

I guess that the new build is bigger because it is much more based on the standard FreeBSD distribution and we get a bunch more standard utilities built in by default.

@Renato:

About the image size, it increased because we are now distributing full packages instead of copy individual files and dependencies based on a copy list.

In the past I've learned (including from these forums) that installing various OS packages on a firewall increases the security risk. How's that statement changing now, with this new approach?

jimp · Oct 5, 2015, 12:08 PM

@robi:

In the past I've learned (including from these forums) that installing various OS packages on a firewall increases the security risk. How's that statement changing now, with this new approach?

We are using the features built into FreeBSD to only build (or exclude) whole areas of the base OS rather than picking and choosing individual files specifically as one done in the past. This brings us closer in parity to FreeBSD which simplifies things overall and makes for a more consistent environment.

robi · Oct 5, 2015, 1:36 PM

I know that, but how will this affect security-wise? A lot of packages will be present there, while in the past they were not… and not even needed for operation.

jimp · Oct 5, 2015, 5:46 PM

@robi:

I know that, but how will this affect security-wise? A lot of packages will be present there, while in the past they were not… and not even needed for operation.

We still don't include a compiler or a compiler environment, and the sort of things we include aren't earth-shattering in terms of security, though we do need to be more careful of what is getting pulled in as dependencies. Before, we had an explicit list of things to copy in, now we take whatever a package claims to need. We have found in some cases that build dependencies were not listed as such, so they were taken as run dependencies as well, so things get added to the pile that aren't really necessary. So we need may need to fix certain ports to behave. As we find them, the image size will go down.

Like right now we're chasing down why perl is in the base system when it shouldn't be, turns out two different ports are claiming it is needed, but we may have to fiddle with some options to pare that down.

Looking at things from the FreeBSD base that are there now that were not there before, mostly it's a few libraries and binaries that we excluded. I don't see anything that wouldn't be useful in some circumstance or another off the top of my head. Some of the libraries we include now will make some packages happier as well.

As with any case when more is added in there is a potential for a security issue, but for the things we're looking at here the impact will be largely minimal.

robi · Oct 5, 2015, 6:01 PM

@jimp:

@robi:

I know that, but how will this affect security-wise? A lot of packages will be present there, while in the past they were not… and not even needed for operation.

We have found in some cases that build dependencies were not listed as such, so they were taken as run dependencies as well, so things get added to the pile that aren't really necessary.

Thanks for the detailed answer.
The qoute above is the reason why I was asking. I see this all the time in Linux too, seems that package devs usually mark as dependencies stuff "just to make sure" without properly checking if it's really needed or not.