force reinstall of packages/OS from the command line?

rcfa

My router was up for over 170 days. I may have updated one or the other package, but that was it. No config changes.
I ended up rebooting it for an unrelated reason, but now it panics on boot.

I can get it to sort-of boot with option 6 at the boot menu, but the web interface won’t come up, and some other things won’t work right, but it’s good enough to access the device with ssh.

Is there a way to force all packages and/or OS to be reinstalled from the command line, given that the most likely reason is a faulty package install? (akin to the Reinstall Packages button on the Diagnostics > Backup & Restore page)

The alternative is to walk a non-technical user over the phone through the reinstallation of pfSense, which isn’t fun for either party involved…
…I do have the config.xml backed up, if worst comes to worst.

jimp

The problem with doing an in-place reinstall with pkg is that it won't address what actually happened there. It could be filesystem corruption in which case it would end up crashing partway through reinstalling things and end up worse off than you are. It could also be the start of a storage failure but in that case you'd be stuck no matter what.

I'd have them download a current installer before you do anything, just in case they have to reinstall anyhow.

If it's running UFS you can start with a Manual Filesystem Check to at least hopefully eliminate that as a contributing factor. If it's running ZFS already then you are more likely looking at a hardware problem if it had an issue like you describe.

After checking the filesystem you can force a reinstall of everything with:

: pkg clean -y
: pkg upgrade -f

The first step empties the pkg cache because if the filesystem had an issue it's possible one of the downloaded packages is broken as well, so that makes it download fresh copies.

It will take a significant amount of time for that to complete in most cases, so be patient.

rcfa

@jimp Thanks! ~~Will try~~ trying that. Best shot I got aside from reinstalling all from scratch anyway…

Oddly enough, the

pkg clean -y

resulted in a

Nothing to do.

response, which was a bit unexpected…
…~~will see how the rest goes.~~here is how the first go-around went, will try a second pass, in case the issues caused were a result of the initial problem. Aside from the messages to be expected, there were a few that “stood out”:

instead of “using existing user…” it said “updating user cyrus”, not sure what there was to update, but harmless enough.
when reinstalling the zeek package, after the line “Extracting zeek-4.0.4: 100%” I got a line that doesn’t sound exactly harmless:

Bad system call

at various points there was a complaint that was odd about about

pfctl: DIOCGETALTQS: Permission denied

But the real kickers when reinstalling pfSense-pkg-arpwatch-0.1.0_6: after the line “Custom commands…” I get:

Executing custom_php_install_command()…fetch: -: Broken pipe
Bad system call (core dumped)
done.
Executing custom_php_resync_config_command()…fetch: -: Broken pipe
Bad system call (core dumped)
done.

Hope it’s not a storage issue, since it’s an SSD which was brand new when installed, is pretty oversized, and the only significant write activity to which is upgrading pfSense, so in terms of life expectancy, it’s well, well below the point where I should expect failure (of course, as they say sh*t happens anyway).
I have a totally non-technical person on the other end, and also only a “metered” iPhone internet connection (without this thing working properly), but I still have the original “credit card” installer USB stick that came with the unit (Netscape Minnowboard based box), so if it comes to that, it’ll be install from scratch from that, put in a pass all rule from the WAN port, and then take over remotely to upgrade the OS to the latest, and then reinstall the backed up config.xml
But as I said, it’s doable, but painful with a non-techie on the other end.

rcfa

@jimp Aside from the inline updates in my original reply, I got an odd response when I tried to run top. Something about swap.
So I issued the swapinfo command, and got this interesting response:

swapinfo: xswdev version mismatch

So something’s messed up, obviously.

Gertjan

@rcfa

Take a couple of minutes.
Take a backup of your config.
Download latest version, and burn it to a USB stick (see pfSense manual).
Install from scratch.

You won't know what happened, but you'll be sure to have a good working system.

jimp

A reinstall is the best course of action given the symptoms. Assuming it's not a hardware problem, that would get it back up and running the fastest.

rcfa

@gertjan Yup, would do that if the unit were local, or had on-location IT staff. Unfortunately talking a non-technical user through that, particularly if the only internet (thanks to this malfunction) is a mobile internet on a phone, makes that a not so great solution.

rcfa

@jimp Ended doing that, by re-installing from the original USB Stick that came with the minnowboard based boxes.
Then creating an easyrule that granted full access from the WAN, and from there I could take over.

Upgraded in two stages to 2.6.0 (first latest 2.4.x version), then switching to the 2.5 branch (which includes seemingly 2.6.0) and updating to that.

However, upon restoring the config.xml, the system got stuck installing packages.
So in the end, I had to delete the package lock, and reinstall all packages manually anyway.

There’s a constant issue with a couple of packages, in that they e.g. wont properly update, they have to be deleted and installed, each time there’s a new version. (squid I think is one of them)

At least things are up again, now. Thanks!