pkg update -f and pkf upgrade -f

NollipfSense

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense care to explain why it is bad? I'm using a PCEngines APU2C4 which only has one slot for an SSD so buying another small 16GB one wouldn't be an option for me.

It can and will fail...your box comes with two external USB 3...get a cheap USB 3 enclosure and put the other SSD in...you're looking $25 for both eBay. Also, I recommend to always have the latest backup and a USB drive with the installer on hand...even ZFS mirror fails or becomes corrupted sometimes...I have had one so far.

kevindd992002

@NollipfSense said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense care to explain why it is bad? I'm using a PCEngines APU2C4 which only has one slot for an SSD so buying another small 16GB one wouldn't be an option for me.

It can and will fail...your box comes with two external USB 3...get an a cheap USB 3 enclosure and put the other SSD in...you're looking $25 for both eBay. Also, I recommend to always have the latest backup and a USB drive with the installer on hand...even ZFS mirror fails or becomes corrupted sometimes...I have had one so far.

How does it "fail"? If I want to stay with ZFS with a single SSD, wold it be better to just not do striping altogether?

bmeeks

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense care to explain why it is bad? I'm using a PCEngines APU2C4 which only has one slot for an SSD so buying another small 16GB one wouldn't be an option for me.

It can and will fail...your box comes with two external USB 3...get an a cheap USB 3 enclosure and put the other SSD in...you're looking $25 for both eBay. Also, I recommend to always have the latest backup and a USB drive with the installer on hand...even ZFS mirror fails or becomes corrupted sometimes...I have had one so far.

How does it "fail"? If I want to stay with ZFS with a single SSD, wold it be better to just not do striping altogether?

If your single SSD fails, what good is ZFS then? The only small improvement ZFS on a single disk has over UFS is ZFS can be slightly more tolerant of abrupt loss of power events.

If you want redundancy, then some type of mirror using separate physical devices is the only way to go. In that scenario, ZFS is a great choice as a resilient file system.

kevindd992002

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@NollipfSense care to explain why it is bad? I'm using a PCEngines APU2C4 which only has one slot for an SSD so buying another small 16GB one wouldn't be an option for me.

It can and will fail...your box comes with two external USB 3...get an a cheap USB 3 enclosure and put the other SSD in...you're looking $25 for both eBay. Also, I recommend to always have the latest backup and a USB drive with the installer on hand...even ZFS mirror fails or becomes corrupted sometimes...I have had one so far.

How does it "fail"? If I want to stay with ZFS with a single SSD, wold it be better to just not do striping altogether?

If your single SSD fails, what good is ZFS then? The only small improvement ZFS on a single disk has over UFS is ZFS can be slightly more tolerant of abrupt loss of power events.

If you want redundancy, then some type of mirror using separate physical devices is the only way to go. In that scenario, ZFS is a great choice as a resilient file system.

Yes, I understand. I mean that's already a given for any type of filesystem. With a single SSD and striping, when the SSD fails then everything fails. It's no different if done without striping, is it? And yes, more tolerance for when there are power loss events is an advantage over UFS. I mean, my pfsense boxes are for home use and it's not like I don't have config backups for them. If ever the SSD fails, I can easily replace, reinstall, and restore.

I'm still wondering how to make two copies of everything in the zsf partition after the reinstall. What does "pkg update -f" and "pkg upgrade -f" really do in this particular scenario?

bmeeks

@kevindd992002
Everything you should need to understand the arguments fo pkg are here. The pkg utility is ignorant of disk partitions themselves. It just references whatever partitions are mounted at the time it runs. pkg is all about simply updating binary and text files on the mounted filesystem. Perhaps you are confusing the pfSense-upgrade program with pkg?

kevindd992002

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002
Everything you should need to understand the arguments fo pkg are here. The pkg utility is ignorant of disk partitions themselves. It just references whatever partitions are mounted at the time it runs. pkg is all about simply updating binary and text files on the mounted filesystem. Perhaps you are confusing the pfSense-upgrade program with pkg?

No. If you read the thread I referenced, they're saying that if you set the zfs copies to 2, only new files will have 2 copies. The existing files will need to be modified for them to be duplicated. Here's the specific post I'm referring to:

https://forum.netgate.com/post/731585

NollipfSense

@kevindd992002 You may need to manually create two GPT partitions of equal size on the single then see whether the installer recognizes both...you'll need to still press the space to select the second partition. You're really on your own...the thread you link in the original provided the how to.

kevindd992002

@NollipfSense said in pkg update -f and pkf upgrade -f:

@kevindd992002 You may need to manually create two GPT partitions of equal size on the single then see whether the installer recognizes both...you'll need to still press the space to select the second partition. You're really on your own...the thread you link in the original provided the how to.

There isn't really a need for two partitions. You simply do zfs set copies=2 yourpoolname to set two copies on the same partition. I just want to know the best way to modify existing files to trigger the process of duplicating themselves to the 2nd zfs copy on the same partition on the same SSD. And then setup a cron schedule to scrub the files every now and then.

NollipfSense

@kevindd992002 So, you're saying you already setup the ZFS on the single and installed pfSense? If that's the case, then run zfs set copies=2 yourpoolname at the CLI, then do the cron job...I take it your pool name is pool1.

kevindd992002

@NollipfSense said in pkg update -f and pkf upgrade -f:

@kevindd992002 So, you're saying you already setup the ZFS on the single and installed pfSense? If that's the case, then run zfs set copies=2 yourpoolname at the CLI, then do the cron job...I take it your pool name is pool1.

Reinstalled pfsense with ZFS
After installation, you have the option to go the shell and set the zfs copies to 2. My pool name is the default zroot.
Since the setting of the copies was done practically after the installation of pfsense, the system files are not duplicated until they get modified.
The cron job is for the scrubbing part and that is not my concern here. My concern is how to get most of the files in the system modified so that zfs creates a duplicate for all of them. As per the post I linked to, he says and I quote:

"Just wait for the next update to your pfSense system, the update will rewrite almost all of the base system files and then you can do the 'pkg upgrade -f' trick to reinstall the rest."

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

NollipfSense

@kevindd992002 It seems to be upgrading a pkg text file in the system; however, I'll link
@stephenw10 in the hope that he could explain better.

bmeeks

@kevindd992002 said in pkg update -f and pkf upgrade -f:

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

Ah, I understand your question a bit better now. All pkg upgrade -f does is force the pkg utility to reinstall all the concurrently installed package files whether they are "current" or not. Normally the utility will only install (or download and write files to disk) when there are newer versions posted on whatever remote repository pkg is pointed at. The "-f" flag just tells pkg to download and rewrite all of the installed package files regardless if they are already current.

So the effect with that switch is everything pkg initially installed is installed again. This would touch all of those files and rewrite their physical content to disk. The normal behavior of pkg upgrade would be to only download and rewrite files to disk when the remote repository contains newer versions of those files. This is explained in that link I provided earlier.

kevindd992002

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

Ah, I understand your question a bit better now. All pkg upgrade -f does is force the pkg utility to reinstall all the concurrently installed package files whether they are "current" or not. Normally the utility will only install (or download and write files to disk) when there are newer versions posted on whatever remote repository pkg is pointed at. The "-f" flag just tells pkg to download and rewrite all of the installed package files regardless if they are already current.

So the effect with that switch is everything pkg initially installed is installed again. This would touch all of those files and rewrite their physical content to disk.

Yeah, I figured that that's what it will do by reading this

Now, as I understand it you also need to update the catalogue before doing an upgrade. That's where pkg update -f comes in. This says that the -f switch will force a full download of the repository. Without it, does it just do a differential download of some sort?

bmeeks

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

Ah, I understand your question a bit better now. All pkg upgrade -f does is force the pkg utility to reinstall all the concurrently installed package files whether they are "current" or not. Normally the utility will only install (or download and write files to disk) when there are newer versions posted on whatever remote repository pkg is pointed at. The "-f" flag just tells pkg to download and rewrite all of the installed package files regardless if they are already current.

So the effect with that switch is everything pkg initially installed is installed again. This would touch all of those files and rewrite their physical content to disk.

Yeah, I figured that that's what it will do by reading this

Now, as I understand it you also need to update the catalogue before doing an upgrade. That's where pkg update -f comes in. This says that the -f switch will force a full download of the repository. Without it, does it just do a differential download of some sort?

Yes, it always attempts to minimize the amount of downloaded/installed data when possible. The -f switch overrides that behavior. If you are familiar with how the Linux package managers work (apt, rpm, etc.) it's the same idea. The normal behavior downloads the smallest bits possible to accomplish the task.

pkg on FreeBSD is the equivalent of apt on Ubuntu.

kevindd992002

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

Ah, I understand your question a bit better now. All pkg upgrade -f does is force the pkg utility to reinstall all the concurrently installed package files whether they are "current" or not. Normally the utility will only install (or download and write files to disk) when there are newer versions posted on whatever remote repository pkg is pointed at. The "-f" flag just tells pkg to download and rewrite all of the installed package files regardless if they are already current.

So the effect with that switch is everything pkg initially installed is installed again. This would touch all of those files and rewrite their physical content to disk.

Yeah, I figured that that's what it will do by reading this

Now, as I understand it you also need to update the catalogue before doing an upgrade. That's where pkg update -f comes in. This says that the -f switch will force a full download of the repository. Without it, does it just do a differential download of some sort?

Yes, it always attempts to minimize the amount of downloaded/installed data when possible. The -f switch overrides that behavior. If you are familiar with how the Linux package managers work (apt, rpm, etc.) it's the same idea. The normal behavior downloads the smallest bits possible to accomplish the task.

pkg on FreeBSD is the equivalent of apt on Ubuntu.

Yes, I do have a Debian system and I use apt update and apt upgrade. The only difference is that they don't have a -f switch which got me confused.

bmeeks

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

@bmeeks said in pkg update -f and pkf upgrade -f:

@kevindd992002 said in pkg update -f and pkf upgrade -f:

The next update will modify/rewrite most of the files, I understand that. What I don't understand is why the need to do a pkg upgrade -f after the upgrade. What does this accomplish?

Ah, I understand your question a bit better now. All pkg upgrade -f does is force the pkg utility to reinstall all the concurrently installed package files whether they are "current" or not. Normally the utility will only install (or download and write files to disk) when there are newer versions posted on whatever remote repository pkg is pointed at. The "-f" flag just tells pkg to download and rewrite all of the installed package files regardless if they are already current.

So the effect with that switch is everything pkg initially installed is installed again. This would touch all of those files and rewrite their physical content to disk.

Yeah, I figured that that's what it will do by reading this

Now, as I understand it you also need to update the catalogue before doing an upgrade. That's where pkg update -f comes in. This says that the -f switch will force a full download of the repository. Without it, does it just do a differential download of some sort?

Yes, it always attempts to minimize the amount of downloaded/installed data when possible. The -f switch overrides that behavior. If you are familiar with how the Linux package managers work (apt, rpm, etc.) it's the same idea. The normal behavior downloads the smallest bits possible to accomplish the task.

pkg on FreeBSD is the equivalent of apt on Ubuntu.

Yes, I do have a Debian system and I use apt update and apt upgrade. The only difference is that they don't have a -f switch which got me confused.

They have a --force switch or something similar. Been a while since I've needed to use either from the command line.

kevindd992002

apt-get does, but not apt. Anyway, will doing a pkg update -f do any potential damage compared to just letting its thing do without the switch?

bmeeks

@kevindd992002 said in pkg update -f and pkf upgrade -f:

apt-get does, but not apt. Anyway, will doing a pkg update -f do any potential damage compared to just letting its thing do without the switch?

No, so long as you have not manually monkeyed with the repo.conf file and say pointed it at some other remote repository that might contain different versions of stuff. But with a stock pfSense installation there is no harm in using the -f switch other than just downloading and rewriting data that strictly does not require such.