"pkg: out of space" on mostly empty CF card
-
@stephenw10 not by choice on this machine. It's remote, and has been reliable for some time (I use ramdisk for /var and /tmp, although I disabled it here in case that was the issue). I'm kind of at a loss as to how to troubleshoot it since I don't normally work with CF-card based systems.
-
There's nothing really very special about CF cards in terms of handling. They are usually very slow because they are IDE and often not even UDMA. They often don't have the same wear leveling which can be an issue.
It's possible the card itself is hitting a wear limit. I would usually recommend installing as UFS with RAM disks when running from an actual flash card to reduce drive writes.
It might not be though. You can try grading from the command line and see if it throws any other errors:
pfSense-upgrade -dIf it is a problem with the card I would expect to see a bunch of drive errors when it hits it. It could be on the console only though if it;s not able to log it.
-
@stephenw10 I tried this, and there were zero errors reported, the process just stopped at the same point. I got onsite and tried a fresh install, and it didn't work either, so I'm assuming the card is cashed and have ordered a new one. If this device had power for a SATA drive I'd swap the CF card for one, but... oh well.
-
Hmm, that's odd failure mode. Usually the drives just stop responding entirely or go read-only. But I'm not sure I've seen a CF card that large.
Anyway I recommend UFS and RAM disks unless you have some packages that don't play nice with ram disks.
-
@stephenw10 Thanks. I had been using ramdisks for /tmp and /var, but disabled them for the upgrade's sake. It's some kind of heavy-duty embedded system one with UDMA7 and everything, but it's likely pretty old at this point so swapping it seems prudent. What's the motivation in this case behind UFS as opposed to ZFS?
-
It's drive writes. ZFS writes a lot more to the disk doing anything. With UFS and ram disks the drive write are close to zero unless you are actively writing out logs etc.
-
Got it. Thanks for the advice!
-
Just an update - it seems like it wasn't the CF card being bad. 24.11 just doesn't like CF cards, from what I can tell - no matter what filesystem or boot options I try, I can't get this thing to boot with a clean install of 24.11. It's an old Lanner 7551 machine, still operational and adequate for what I had it doing, so I'm going to futz with getting a SATA drive in there (it needs a floppy-to-SATA-power cable to use one) and see if I can get it running on a SATA SSD. If not, it's over 10 years old, so I don't feel too bad putting it to pasture, but I hate wasting hardware :)
-
Hmm, I have one of those, let me see.....
-
Hmm, well it failed to upgrade the first time I tried it but succeeded when upgrading at the console. From 23.09.1 to 24.11.
That's a UFS install to a 4GB CF card. Not something I would recommend because it is very slow!
However it does boot fine. I'm wondering whether your much large CF card is hitting some limit....
0x8+0x1d4108+0x8+0x1e9a19/c808+0x3b37f8 Loading configured modules...___ ___ _ /boot/entropy size=0x1000_ \/ __|/ _ \ _| |_ /boot/kernel/wbwd.ko size 0x45b8 at 0x37bf000_| loading required module 'superio'\___| |_| /boot/kernel/superio.ko size 0x6df8 at 0x37c4000 can't find '/etc/hostid' GDB: no debug ports present KDB: debugger backends: ddbfSense Plus ----+ __________________________ KDB: current backend: ddb | / ___\ ---<<BOOT>>---lti user [Enter] | | /` Copyright (c) 1992-2024 The FreeBSD Project. | / :-| Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 /_ | | 4. RThe Regents of the University of California. All rights reserved.__/ | FreeBSD is a registered trademark of The FreeBSD Foundation. / / / | FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:08:46 UTC 2024 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/amd64/AKWlAIiM/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBSD-src-plus-RELENG_24_11/amd64.amd64/sys/pfSense amd64 / _| |_ | FreeBSD clang version 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67) VT(vga): resolution 640x480 | / |_| | CPU: Intel(R) Atom(TM) CPU C2358 @ 1.74GHz (1750.07-MHz K8-class CPU)_____/ Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 - Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch>xt=0xff3048 / Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 4039811072 (3852 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <INTEL TIANO > WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 <Version 2.0> irqs 0-23 Launching APs: 1 TCP_ratelimit: Is now initialized ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807543a0, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80754450, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80754500, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80773fb0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80774060, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80774110, 0) error 1 random: entropy device external interface wlan: mac acl policy registered kbd1 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0. netgate0: <unknown hardware> netgate0: version: 0.1 vtvga0: <VT VGA driver> smbios0: <System Management BIOS> at iomem 0xf0570-0xf058e smbios0: Version: 2.8, BCD Revision: 2.7 acpi0: <ALASKA A M I > acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> mem 0xdff00000-0xdff1ffff irq 16 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> mem 0xdfee0000-0xdfefffff irq 16 at device 2.0 on pci0 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> mem 0xdfec0000-0xdfedffff irq 20 at device 3.0 on pci0 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> mem 0xdfea0000-0xdfebffff at device 4.0 on pci0 pci4: <ACPI PCI bus> on pcib4 pci0: <processor> at device 11.0 (no driver attached) pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached) igb0: <Intel(R) I354 (SGMII)> port 0xf0c0-0xf0df mem 0xdfe60000-0xdfe7ffff,0xdff2c000-0xdff2ffff irq 20 at device 20.0 on pci0 igb0: EEPROM V1.5-0 eTrack 0x80000ba5 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 2 RX queues 2 TX queues igb0: Using MSI-X interrupts with 3 vectors igb0: Ethernet address: 00:90:0b:38:ef:a0 igb0: netmap queues/slots: TX 2/1024, RX 2/1024 igb1: <Intel(R) I354 (SGMII)> port 0xf0a0-0xf0bf mem 0xdfe40000-0xdfe5ffff,0xdff28000-0xdff2bfff irq 21 at device 20.1 on pci0 igb1: EEPROM V1.5-0 eTrack 0x80000ba5 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: Using 2 RX queues 2 TX queues igb1: Using MSI-X interrupts with 3 vectors igb1: Ethernet address: 00:90:0b:38:ef:a1 igb1: netmap queues/slots: TX 2/1024, RX 2/1024 igb2: <Intel(R) I354 (SGMII)> port 0xf080-0xf09f mem 0xdfe20000-0xdfe3ffff,0xdff24000-0xdff27fff irq 22 at device 20.2 on pci0 igb2: EEPROM V1.5-0 eTrack 0x80000ba5 igb2: Using 1024 TX descriptors and 1024 RX descriptors igb2: Using 2 RX queues 2 TX queues igb2: Using MSI-X interrupts with 3 vectors igb2: Ethernet address: 00:90:0b:38:ef:a2 igb2: netmap queues/slots: TX 2/1024, RX 2/1024 igb3: <Intel(R) I354 (SGMII)> port 0xf060-0xf07f mem 0xdfe00000-0xdfe1ffff,0xdff20000-0xdff23fff irq 23 at device 20.3 on pci0 igb3: EEPROM V1.5-0 eTrack 0x80000ba5 igb3: Using 1024 TX descriptors and 1024 RX descriptors igb3: Using 2 RX queues 2 TX queues igb3: Using MSI-X interrupts with 3 vectors igb3: Ethernet address: 00:90:0b:38:ef:a3 igb3: netmap queues/slots: TX 2/1024, RX 2/1024 ehci0: <Intel Avoton USB 2.0 controller> mem 0xdff37000-0xdff373ff irq 23 at device 22.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 ahci0: <Intel Avoton AHCI SATA controller> port 0xf150-0xf157,0xf140-0xf143,0xf130-0xf137,0xf120-0xf123,0xf040-0xf05f mem 0xdff36000-0xdff367ff irq 19 at device 23.0 on pci0 ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahci1: <Intel Avoton AHCI SATA controller> port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xdff35000-0xdff357ff irq 19 at device 24.0 on pci0 ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported ahcich2: <AHCI channel> at channel 0 on ahci1 ahcich3: <AHCI channel> at channel 1 on ahci1 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 apei0: <ACPI Platform Error Interface> on acpi0 ns8250: UART FCR is broken ns8250: UART FCR is broken uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] superio0: <Winbond 83627DHG-P> at port 0x2e-0x2f on isa0 wbwd0: <Winbond 83627DHG-P (0xb0/0x73) Watchdog Timer> at WDT ldn 0x08 on superio0 ns8250: UART FCR is broken ns8250: UART FCR is broken uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 Timecounter "TSC" frequency 1749999951 Hz quality 1000 Timecounters tick every 1.000 msec ugen0.1: <Intel EHCI root HUB> at usbus0 uhub0 on usbus0 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 Trying to mount root from ufs:/dev/ufsid/5cad271f6a7f8993 [rw]... Root mount waiting for: usbus0 CAM Root mount waiting for: usbus0 CAM Root mount waiting for: usbus0 CAM uhub0: 8 ports with 8 removable, self powered ugen0.2: <vendor 0x8087 product 0x07db> at usbus0 uhub1 on uhub0 uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0 Root mount waiting for: usbus0 CAM uhub1: 4 ports with 4 removable, self powered Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <CF 4GB 20110221> ATA-7 SATA 1.x device ada0: Serial Number TSS50034141104223548 ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 512bytes) ada0: 3831MB (7847280 512 byte sectors) Dual Console: Serial Primary, Video Secondary Configuring crash dumps... No suitable dump device was found. /dev/ufsid/5cad271f6a7f8993: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufsid/5cad271f6a7f8993: clean, 228868 free (2300 frags, 28321 blocks, 0.3% fragmentation) Filesystems are clean, continuing... Mounting filesystems... Setting up memory disks... done. Creating pkg db symlink Creating pkg cache symlink Restoring contents of RAM disk store... done. __ _ __ / _|___ ___ _ __ ___ ___ _ | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \ _| |_ | |_) | _\__ \ __/ | | \__ \ __/ |_ _| | .__/|_| |___/\___|_| |_|___/\___| |_| |_| Welcome to Netgate pfSense Plus 24.11-RELEASE... ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.36/mach/CORE 32-bit compatibility ldconfig path: done. 3513 >>> Removing vital flag from php83...done. External config loader 1.0 is now starting... ada0s1 ada0s1a Launching the init system...Updating CPU Microcode... CPU: Intel(R) Atom(TM) CPU C2358 @ 1.74GHz (1750.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> Structured Extended Features3=0xc000400<MD_CLEAR,IBPB,STIBP> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics Done. done. Initializing.................... done. Starting device manager (devd)...ichsmb0: <Intel Avoton SMBus controller> port 0xf000-0xf01f mem 0xdff34000-0xdff3401f irq 18 at device 31.3 on pci0 smbus0: <System Management Bus> on ichsmb0 done. Loading configuration...done. Updating configuration...done. Loading cryptographic accelerator drivers...aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> done. Setting up extended sysctls...done. Executing early shell commands...done. coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Setting timezone...done. Configuring looplo0: link state changed to UP back interface...done. Starting syslog...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Configuring WAN interface...done. Configuring LAN interface...done. Configuring CARP settings...done. Starting Secure Shell Services...done. Syncing OpenVPN settings...done. Configuring firewall......done. Starting PFLOG...done. Setting up gateway monitors...done. Setting up static routes...done. Setting up DNSs... Starting DNS Resolver...done. Synchronizing user settings...done. Configuring CRON...done. Bootstrapping clock...done. Starting NTP Server...done. Starting webConfigurator...done. Starting DHCP service...done. Configuring firewall......done. Generating RRD graphs...done. Starting syslog...done. Starting CRON... done. Netgate pfSense Plus 24.11-RELEASE amd64 20241127-1822 Bootup complete FreeBSD/amd64 (7551.stevew.lan) (ttyu0) Netgate pfSense Plus - Netgate Device ID: 692b83cd6122f2xxxxxx *** Welcome to Netgate pfSense Plus 24.11-RELEASE (amd64) on 7551 *** WAN (wan) -> igb0 -> v4/DHCP4: 172.21.16.18/24 LAN (lan) -> igb1 -> v4: 192.168.130.1/24 0) Logout / Disconnect SSH 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart GUI 3) Reset admin account and password 12) PHP shell + Netgate pfSense Plus tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option: -
@stephenw10 that's interesting. This unit has had a 64GB CF card in it for ~10 years and only had an issue now, but even when I swapped it for a brand new one and did a fresh install of 24.11, it still had the same issue. I know back in the day we needed to use a specific image to install on CF, I wonder if I'm hitting a corner case of 1) the original CF card began to fail and 2) the latest installer doesn't know how to do CF geometry for booting anymore? Because if I install with MBR (using UFS or ZFS), it gets to the loader that offers the F1, F2, and F6 options, but won't go beyond that. GPT partitioning doesn't get that far, but I seem to remember using it before (and the BIOS offers the option to drop into an EFI shell, which would say to me it does at least support GPT booting).
-
Yeah it feels like it might be an edge case you're hitting there. Almost no-one will be using a CF that size. I don't think we ever shipped anything larger than 8GB.
Also I'm almost certain that CF card is not running at the UDMA5 it reports. Almost all CF slots were missing the pins to actually enable UDMA. Way back when the driver used to show that.
-
hey. i got a similar issue by upgrading 24.05 > 24.11 . after deleting unused boot environment it upgraded smoothly.
-
Yes, if you're actually out of space because the drive is filled with BE snaps it will also show that error.
-
The C2k Intel Aroms are run into some LPC problems and die.
But it start strange, some can be recovered with a 100Ohm resistor. -
But that's unrelated to boot media. I would not expect that to ever present like this.
-
I didn't have any unused BEs or anything, the card was almost entirely clear. It was something else. Also not the C2k bug, I've dealt with over a hundred machines with that, and rescued most with the resistor trick :) I'm hoping to give the machine another go with a SATA drive tomorrow, I finally found the floppy->SATA power cable needed for this weird little board.
-
I found the problem - I tried reinstalling with an archived copy of the old standalone 24.03 installer, selected BIOS+Active, and boom - booted. The new installer doesn't have the ability to boot these older boxes. So I was running into a cascade of issues - first, my CF card failed, then the installer wasn't able to set up the hardware to boot properly.
There might be other options the old installer could do that let this machine boot - I know there were a bunch of options - I wonder if they could be added back to the new installer to give us the ability to keep this otherwise in-spec hardware running.
-
Also, forgot to mention, it upgraded to 24.11 no problem :)
-
Hmm, so the net installer doesn't boot at all? IIRC there were some quirks on those older boxes to allow it to boot.
