6100 failing after trying to upgrade from 23.09.01-RELEASE to 24.xx

cedrictang

I don't want to take away the efforts of the support team. They have been helpful. But I won't want to let this one go because it is still a 700 USD device.

I tried

FreeBSD-14.1-BETA3-amd64-disc1.iso
pfSense-plus-memstick-serial-24.03-RELEASE-amd64.img.gz

Used both Rufus and Etcher to write to 2 different USB sticks (one 3.0 and one 3.2 gen1).

When I got to the screen to press r pr p or f2, the next blue menu only offers the NVME drive or PXE.

I could get to a PFSense Plus menu to boot either in multi user or single user or escape to loader prompt etc. I was hoping the loader prompt might be of help.

If I leave it to boot further I eventually get to the login. But there are a tons of fatal error messages before that.

On that screen I cannot do anything (eg change interface config etcetc). The fatal error comes up. I can only get to the shell with option 8. The device is still on 23.09.01.

Welcome to Netgate pfSense Plus 23.09.1-RELEASE...

Checking dump device /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6 for crash dumps ... no crash dumps on /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
7552

Removing vital flag from php83...done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/rc.ecl(25): require_once('/etc/inc/functi...')
#2 {main}
thrown in /etc/inc/certs.inc on line 48
Launching the init system...
Fatal error: Uncaught Error: Failed opening required 'Net/IPv6.php' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /etc/inc/util.inc:28
Stack trace:
#0 /etc/inc/config.lib.inc(29): require_once()
#1 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#2 /etc/inc/captiveportal.inc(29): require_once('/etc/inc/auth.i...')
#3 /etc/inc/service-utils.inc(25): require_once('/etc/inc/captiv...')
#4 /etc/inc/pkg-utils.inc(26): require_once('/etc/inc/servic...')
#5 /etc/rc.bootup(27): require_once('/etc/inc/pkg-ut...')
#6 {main}
thrown in /etc/inc/util.inc on line 28
Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(25): require_once('/etc/inc/config...')
#4 {main}
thrown in /etc/inc/certs.inc on line 48
Netgate pfSense Plus 23.09.1-RELEASE amd64 20240404-1554
Bootup complete
)FreeBSD/amd64 (Amnesiac) (ttyu0)

login:

stephenw10

Yup even with those errors you should still be able to reach a shell. I would try just connecting the USB drives at that point to see if the OS recognises them at all. If not it may actually be bad USB ports somehow.

cedrictang

@stephenw10

Thank you Stephen. Again I don't have the linux skills to diagnose the problem. If you have any pointers that would be most appreciated.

(I am only able to set things up by following perhaps youtube videos and my average computer knowledge)

cedrictang

@stephenw10

Yes I can get to the shell after the Amnesiac prompt. Can login with admin user and the password.
I can do the basic commands looking at the partition table, tops. etcetc.

stephenw10

Ok so first try imply plugging in the USB stick. At the serial console like that you should see the new drive reported directly like:

 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: ugen0.4: <TOSHIBA TransMemory> at usbus0
umass1 on uhub1
umass1: <TOSHIBA TransMemory, class 0/0, rev 2.00/1.00, addr 4> on usbus0
da1 at umass-sim1 bus 1 scbus7 target 0 lun 0
da1: <TOSHIBA TransMemory 1.00> Removable Direct Access SPC-2 SCSI device
da1: Serial Number 0022CFF6B899C34162869AF8
da1: 40.000MB/s transfers
da1: 14782MB (30274560 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

cedrictang

@stephenw10

No nothing comes up. In either of the usb slots. But the usb stick does get hot (so it is getting some power).

stephenw10

Ok try running usbconfig. That will show all the devices including the host hubs But you should be able to clearly see the drive when it's connected.

[24.03-RELEASE][admin@4200.stevew.lan]/root: usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen1.1: <Intel XHCI root HUB> at usbus1, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen1.2: <Generic Ultra Fast Media> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (96mA)
ugen1.3: <TOSHIBA TransMemory> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (200mA)

cedrictang

@stephenw10

I only have

ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)

despite having 2 sticks attached to both ports.

But I just tried to plug a usb mouse into the bottom port. Maybe this is a worrying sign because when fully plugged in the red mouse movement light does not come on but if i pull it out 3mm, it does! So maybe the port is broken?!!

stephenw10

Hmm, that's not a good sign! Does it look physically damaged? Does the mouse appear in the usbconfig list when it's partially removed like that?

cedrictang

@stephenw10

Doesn't look damaged. And seriously I had never needed to use those ports in the first place. And no the mouse does not show up either.

stephenw10

Hmm, try using a USB extension cable if you can or a hub. See if that makes any difference at all.

One possibility would be to write the SSD in something else and transfer it back. A bit round about but it would get the device up again.

Another option might be to write the install image to the eMMC and then boot that. To do that though you would need some network connectivity to get the image.

cedrictang

@stephenw10

Thanks. I will give it the extension cable a try.
I need to get this replaced to bring the network back up to its original state.
Will try to find a way to see how one can hack this.

Maybe need to take the NVME drive out. The device is out of warranty anyway.

Or maybe boot from the network somehow using PXE.

This will be another project.

Separately, I did think about some HA solution before after the last episode. Unfortunately I need to use dual WAN because my site has patchy DSL. I haven't figured how to do HA yet. Maybe one unit on dual WAN and another unit on a 4G single WAN. Any suggestions?
Or I could get by with 2 units with a single WAN connection on each.

stephenw10

Just to confirm you are not running ZFS there currently?

Because if you are you should just be able to roll back the snapshot from the loader menu and try the upgrade again.
https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/loader.html

If you setup a hardware HA pair of devices they should be as close to identical as possible. That means both nodes should have access to both WANs ideally. That gets complex because to do that you need multiple public IP addresses on each WAN.

Running a single, separate WAN to each node is not really a valid setup. Though I have seen it done.

Steve

cedrictang

@stephenw10 said in 6100 failing after trying to upgrade from 23.09.01-RELEASE to 24.xx:

https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/loader.html

Thanks for your message!
I don't see an option 8 during boot up.
So I guess it is not ZFS. I might recall it is UFS???!!

stephenw10

Urgh, unfortunate. That device might be old enough to have had UFS installed by default.

cedrictang

@stephenw10

bought in dec 2021 when this first came out.

stephenw10

Hmm, well if you can reach a prompt and get network connectivity you could try uploading an image and writing it to the eMMC. Otherwise I thing writing the SSD in something else might be the only option.

cedrictang

@stephenw10

wow so I am real chuffed!

bought a ryzen mini pc and replace the original nvme with the stick from the 6100. booted off the usb. initially the system will boot until a point where it will just halt. it didn't recognised the 2 realtek nics. plugged in a usb network port and the whole thing then booted up correctly and i managed to reinstall the image.

reinstalled the stick back into the 6100 and connected to the console via putty and micro-usb and reconfigured the WAN/LAN and managed to get into the web interface. took about 1.5 hours of figuring out the various dead-ends.

stephenw10

Nice! So it really is just the USB port/bus that failed.

cedrictang

@stephenw10

well yes that. Plus ultimately the upgrade from 23.09 to 24.xx killed the OS which bricked the device.