/libexec/ld-elf.so.1: /usr/local/lib/libiconv.so.3: unsupported file layout

rcfa

Note: this is about an amd64 configuration…

For the sake of testing I installed just about all packages available (as I mentioned elsewhere), and things seemed to have worked just fine, or so I thought.

Then I had to shut down the system because I needed to plug it into another outlet.
Now, when I boot, the web configurator will not launch, and the system isn't even pingable.
So I attached a monitor and keyboard (not the normal configuration of a Lanner 7535 unit...) and the boot gets up to the pfSense console menu. But choosing any option there besides breaking into the shell, gives me the error message listed in the subject:

Safe boot didn't do anything to change the situation, and trying to use option 13 (Upgrade from console), also throws this error message, is thus a non-option.

Now here the questions:
a) what caused this? Is this a known issue I overlooked?
b) how do I fix it, short of completely wiping the volume and reinstalling from scratch?

A general comment: I wish there were something in place that could prevent any build or package from being downloadable if there is any issue with installation that could result in a system no longer accessible from the net. For something that might end up in an embedded system that's remotely managed, a situation like this would be a major disaster. I know we're still testing, but I think that category of bugs should get special attention, particularly the question how to prevent things like these for official releases and packages available from the server.
Other bugs are much less critical: just install an update that fixes the bug, but if the system spins out of control like this, updates are no longer an option, not even slogin is an option currently for this machine.

jimp

Something like that would really be hard to say for certain without trying each package individually to see which produces the error.

When we find problems like that, we can disable a package so it doesn't show up in the list, but unless we know it has a problem, we don't know to disable it, of course.

rcfa

Obviously, we're still testing, so this sort of thing (unfortunately) has to be expected.
However, I would say once deployment phase starts, installing all packages and doing upgrades, or installing all packages after a new install should be part of the testing routine, before a new update is officially released.

On the other hand: is this situation I have here recoverable from the shell, or do I have to re-install from scratch.

Since none of the menu commands, except for breaking to the shell, work, and that only at the console level, it seems that the entire setup was hosed beyond recovery, or at least beyond recovery that requires console access and an external drive from which to try to copy the file in question.

jimp

You probably will need to reinstall from CD, though you could upload a firmware .tgz to the box via scp (Or fetch it) and then unpack it by hand to avoid the PHP code.

rcfa

This brings me to another idea: it might be a really cool thing to have a base-line distribution in a .tgz file along with some shell script that could be invoked to do disaster recovery for that sort of thing. Obviously we don't have a 2.0 release yet, but in the future, that baseline could be a 2.0 release. So if any point upgrade gives trouble, one could attach an external keyboard and blindly type in a (hopefully simple) key sequence, and the box could acknowledge with a beep that it reverts to the baseline setup.

Important would only be that the basic IP configuration and the anti-lockout rule remain intact. Then the box would be back under control.

The key question for me is always: how to recover a box that's half a continent away from me, and doesn't have a keyboard and monitor attached. (Keyboard can be attached relatively easily if need, be. Monitor is a different story…)

jimp

See /etc/rc.create_full_backup and /etc/rc.restore_full_backup

Beyond that keeping a "baseline" isn't really too feasible for a number of reasons, but that would get you close.

As for a box that is remote, that's where having a CARP cluster comes in handy, as does having them hooked up to another box via serial console. (Or if each unit in a CARP cluster has two serial ports, they can buddy up with each other and connect to the partner's serial console)

rcfa

Unfortunately, two boxes won't be in my budget. That aside: would it be possible to have the serial ports reconfigured based on the CARP status, i.e. the one that's live becomes the terminal side, and the one that's down is the one that outputs the console to the serial, and vice versa? Then only one serial port each would be required.

jimp

No, serial console is always active and can't be flipped on or off like that. It's set at bootup.