6100 failing after trying to upgrade from 23.09.01-RELEASE to 24.xx

cedrictang

Bad mistake trying to upgrade this last night causing the 6100 max to basically stop working.

Netgate support is trying to help me and has been helpful but I am unable to recover from this yet.

So posting here to see any other suggestions.

Connected via the console. 23.09.01 boots but get this error:

Welcome to Netgate pfSense Plus 23.09.1-RELEASE...

Checking dump device /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6 for crash dumps ... no crash dumps on /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
7546

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/rc.ecl(25): require_once('/etc/inc/functi...')
#2 {main}
thrown in /etc/inc/certs.inc on line 48
Launching the init system...
Fatal error: Uncaught Error: Failed opening required 'Net/IPv6.php' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /etc/inc/util.inc:28
Stack trace:
#0 /etc/inc/config.lib.inc(29): require_once()
#1 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#2 /etc/inc/captiveportal.inc(29): require_once('/etc/inc/auth.i...')
#3 /etc/inc/service-utils.inc(25): require_once('/etc/inc/captiv...')
#4 /etc/inc/pkg-utils.inc(26): require_once('/etc/inc/servic...')
#5 /etc/rc.bootup(27): require_once('/etc/inc/pkg-ut...')
#6 {main}
thrown in /etc/inc/util.inc on line 28
Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(25): require_once('/etc/inc/config...')
#4 {main}
thrown in /etc/inc/certs.inc on line 48
Netgate pfSense Plus 23.09.1-RELEASE amd64 20240404-1554
Bootup complete
)FreeBSD/amd64 (Amnesiac) (ttyu0)

login:

Netgate support tried to suggest I reinstall via usb. However the 6100 doesn't see the USB stick (tried both the offline 24.03 installer as well as the netinstaller). Tried both ports and 2 separate usb 3.2 drives. No luck.
The bootloader i guess only detects the NVME drive and PXE.

So I am waiting for Netgate support to offer alternative solution. My other Netgate 6100 max also failed before with upgrade 4 months ago in January when I tried to upgrade to 23.09. But the USB boot worked then and I was able to reinstall.

stephenw10

@cedrictang said in 6100 failing after trying to upgrade from 23.09.01-RELEASE to 24.xx:

However the 6100 doesn't see the USB stick

It's not shown in the boot device menu (F2)?

How are you you writing the image to the stick?

cedrictang

@stephenw10

No the 6100 cannot see the usb stick. Tried 2 separate USB drives and both slots.
Boot option only gives the NVME and PXE.

So based on the past 24h, the conclusion is that this device is essentially bricked.

Though the support team has been v responsive on the email and very helpful, a basic upgrade caused this 2 ½ year old 700 USD device to become a brick. I don't have the necessary skills to repair what I would presume to be data corruption and I am unable to effect a reinstall.

I encountered a similar problem just 4 months ago with another 6100 MAX on the 23.09 upgrade. Luckily that time I was able to reinstall and download the cloud config.

stephenw10

Hmm, I've never seen a 6100 not see USB drive at all as long as it's bootable.

Exactly which image did you use and how did you write it to the drive?

Did you try clearing the boot device list?

Are you able to login at that Amnesiac prompt? Probably admin:root or root:root if it really can't find a config.

cedrictang

@stephenw10

I don't want to take away the efforts of the support team. They have been helpful. But I won't want to let this one go because it is still a 700 USD device.

I tried

FreeBSD-14.1-BETA3-amd64-disc1.iso
pfSense-plus-memstick-serial-24.03-RELEASE-amd64.img.gz

Used both Rufus and Etcher to write to 2 different USB sticks (one 3.0 and one 3.2 gen1).

When I got to the screen to press r pr p or f2, the next blue menu only offers the NVME drive or PXE.

I could get to a PFSense Plus menu to boot either in multi user or single user or escape to loader prompt etc. I was hoping the loader prompt might be of help.

If I leave it to boot further I eventually get to the login. But there are a tons of fatal error messages before that.

On that screen I cannot do anything (eg change interface config etcetc). The fatal error comes up. I can only get to the shell with option 8. The device is still on 23.09.01.

Welcome to Netgate pfSense Plus 23.09.1-RELEASE...

Checking dump device /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6 for crash dumps ... no crash dumps on /dev/gptid/f311d7c6-58f5-11ec-ab14-90ec772123d6.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.36/mach/CORE
32-bit compatibility ldconfig path:
done.
7552

Removing vital flag from php83...done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/rc.ecl(25): require_once('/etc/inc/functi...')
#2 {main}
thrown in /etc/inc/certs.inc on line 48
Launching the init system...
Fatal error: Uncaught Error: Failed opening required 'Net/IPv6.php' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /etc/inc/util.inc:28
Stack trace:
#0 /etc/inc/config.lib.inc(29): require_once()
#1 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#2 /etc/inc/captiveportal.inc(29): require_once('/etc/inc/auth.i...')
#3 /etc/inc/service-utils.inc(25): require_once('/etc/inc/captiv...')
#4 /etc/inc/pkg-utils.inc(26): require_once('/etc/inc/servic...')
#5 /etc/rc.bootup(27): require_once('/etc/inc/pkg-ut...')
#6 {main}
thrown in /etc/inc/util.inc on line 28
Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(25): require_once('/etc/inc/config...')
#4 {main}
thrown in /etc/inc/certs.inc on line 48
Netgate pfSense Plus 23.09.1-RELEASE amd64 20240404-1554
Bootup complete
)FreeBSD/amd64 (Amnesiac) (ttyu0)

login:

stephenw10

Yup even with those errors you should still be able to reach a shell. I would try just connecting the USB drives at that point to see if the OS recognises them at all. If not it may actually be bad USB ports somehow.

cedrictang

@stephenw10

Thank you Stephen. Again I don't have the linux skills to diagnose the problem. If you have any pointers that would be most appreciated.

(I am only able to set things up by following perhaps youtube videos and my average computer knowledge)

cedrictang

@stephenw10

Yes I can get to the shell after the Amnesiac prompt. Can login with admin user and the password.
I can do the basic commands looking at the partition table, tops. etcetc.

stephenw10

Ok so first try imply plugging in the USB stick. At the serial console like that you should see the new drive reported directly like:

 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: ugen0.4: <TOSHIBA TransMemory> at usbus0
umass1 on uhub1
umass1: <TOSHIBA TransMemory, class 0/0, rev 2.00/1.00, addr 4> on usbus0
da1 at umass-sim1 bus 1 scbus7 target 0 lun 0
da1: <TOSHIBA TransMemory 1.00> Removable Direct Access SPC-2 SCSI device
da1: Serial Number 0022CFF6B899C34162869AF8
da1: 40.000MB/s transfers
da1: 14782MB (30274560 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

cedrictang

@stephenw10

No nothing comes up. In either of the usb slots. But the usb stick does get hot (so it is getting some power).

stephenw10

Ok try running usbconfig. That will show all the devices including the host hubs But you should be able to clearly see the drive when it's connected.

[24.03-RELEASE][admin@4200.stevew.lan]/root: usbconfig
ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen1.1: <Intel XHCI root HUB> at usbus1, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen1.2: <Generic Ultra Fast Media> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (96mA)
ugen1.3: <TOSHIBA TransMemory> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (200mA)

cedrictang

@stephenw10

I only have

ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)

despite having 2 sticks attached to both ports.

But I just tried to plug a usb mouse into the bottom port. Maybe this is a worrying sign because when fully plugged in the red mouse movement light does not come on but if i pull it out 3mm, it does! So maybe the port is broken?!!

stephenw10

Hmm, that's not a good sign! Does it look physically damaged? Does the mouse appear in the usbconfig list when it's partially removed like that?

cedrictang

@stephenw10

Doesn't look damaged. And seriously I had never needed to use those ports in the first place. And no the mouse does not show up either.

stephenw10

Hmm, try using a USB extension cable if you can or a hub. See if that makes any difference at all.

One possibility would be to write the SSD in something else and transfer it back. A bit round about but it would get the device up again.

Another option might be to write the install image to the eMMC and then boot that. To do that though you would need some network connectivity to get the image.

cedrictang

@stephenw10

Thanks. I will give it the extension cable a try.
I need to get this replaced to bring the network back up to its original state.
Will try to find a way to see how one can hack this.

Maybe need to take the NVME drive out. The device is out of warranty anyway.

Or maybe boot from the network somehow using PXE.

This will be another project.

Separately, I did think about some HA solution before after the last episode. Unfortunately I need to use dual WAN because my site has patchy DSL. I haven't figured how to do HA yet. Maybe one unit on dual WAN and another unit on a 4G single WAN. Any suggestions?
Or I could get by with 2 units with a single WAN connection on each.

stephenw10

Just to confirm you are not running ZFS there currently?

Because if you are you should just be able to roll back the snapshot from the loader menu and try the upgrade again.
https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/loader.html

If you setup a hardware HA pair of devices they should be as close to identical as possible. That means both nodes should have access to both WANs ideally. That gets complex because to do that you need multiple public IP addresses on each WAN.

Running a single, separate WAN to each node is not really a valid setup. Though I have seen it done.

Steve

cedrictang

@stephenw10 said in 6100 failing after trying to upgrade from 23.09.01-RELEASE to 24.xx:

https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/loader.html

Thanks for your message!
I don't see an option 8 during boot up.
So I guess it is not ZFS. I might recall it is UFS???!!

stephenw10

Urgh, unfortunate. That device might be old enough to have had UFS installed by default.

cedrictang

@stephenw10

bought in dec 2021 when this first came out.