Out of disk during upgrade to 23.09, system boots (badly) but I can ssh in...
-
I have two routers running CARP and so it's normal for me to, when I notice a new version available, upgrade one and then the other. It's a good regular test of redundancy...
This time my Alix APU1D upgraded just fine, but my Fujitsu Futro S720 hung during the upgrade reboot.
I have serial access to the device and so can login (as root, it isn't booting pfsense) to the console.
Can anyone suggest what I can do to get this back up running? I have rescued config.xml (by capturing my serial session) so a worst-case option is to reinstall and reconfigure. I'd like to avoid that if possible. I'm hoping there is a "delete this directory and reboot so the upgrade will complete" type of option.
Details follow.
-
I have no network, and here is a boot...
/ __ _ __ / _|___ ___ _ __ ___ ___ _ | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \ _| |_ | |_) | _\__ \ __ +---- Welcome to Netgate pfSense Plus ----+ __________________________ | | / ___\ | 1. Boot Multi user [Enter] | | /` | 2. Boot Single user | | / :-| | 3. Escape to loader prompt | | _________ ___/ /_ | | 4. Reboot | | /` ____ / /__ ___/ | | 5. Cons: Dual (Serial primary) | | / / / / / / | | | | / /___/ / / / | | Options: | | / ______/ / / _ | | 6. Kernel: default/kernel (1 of 2) | |/ / / / _| |_ | | 7. Boot Options | / /___/ |_ _| | | | / |_| | | | /_________________________/ +-----------------------------------------+ - Autoboot in 0 seconds. [Space] to pause Loading kernel... /boot/kernel/kernel text=0x19c338 text=0xfe1270 text=0x17dda48 data=0x180 data=0x223710+0x3db8f0 0x8+0x1c8828+0x8+0x1d6fad/ Loading configured modules... /boot/entropy size=0x1000 /etc/hostid size=0x25 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<<BOOT>>--- Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT amd64 1400094 #0 plus-RELENG_23_09-n256163-2763857e770: Wed Nov 1 21:18:24 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/WrS3lKLo/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/sources/FreeBSD-src-plus-RELENG_23_09/amd64.amd64/sys/pfSense amd64 FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) VT(vga): resolution 640x480 CPU: AMD GX-217GA SOC with Radeon(tm) HD Graphics (1646.99-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x700f01 Family=0x16 Model=0x0 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x154037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PL2I> Structured Extended Features=0x8<BMI1> XSAVE Features=0x1<XSAVEOPT> SVM: NP,NRIP,AFlush,DAssist,NAsids=8 TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 3470401536 (3309 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <FTS D3313-B1> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) random: unblocking device. Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20221020/tbfadt-796) ioapic0 <Version 2.1> irqs 0-23 ioapic1 <Version 2.1> irqs 24-55 Launching APs: 1 TCP_ratelimit: Is now initialized iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8076b040, 0) error 1 random: entropy device external interface wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8074d300, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8074d3b0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8074d460, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8076aee0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8076af90, 0) error 1 kbd1 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0. vtvga0: <VT VGA driver> netgate0: <unknown hardware> smbios0: <System Management BIOS> at iomem 0xf04c0-0xf04de smbios0: Version: 2.8, BCD Revision: 2.7 acpi0: <FTS D3313-B1> acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xd0000000-0xd07fffff,0xfeb00000-0xfeb3ffff irq 44 at device 1.0 on pci0 vgapci0: Boot video device hdac0: <ATI Kabini HDA Controller> mem 0xfeb64000-0xfeb67fff irq 45 at device 1.1 on pci0 pcib1: <ACPI PCI-PCI bridge> irq 29 at device 2.2 on pci0 pci1: <ACPI PCI bus> on pcib1 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xfea00000-0xfea00fff,0xd0a00000-0xd0a03fff irq 28 at device 0.0 on pci1 re0: Using 1 MSI-X message re0: ASPM disabled re0: Chip rev. 0x4c000000 re0: MAC rev. 0x00000000 miibus0: <MII bus> on re0 rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 90:1b:0e:5f:43:da re0: netmap queues/slots: TX 1/256, RX 1/256 pcib2: <ACPI PCI-PCI bridge> irq 28 at device 2.5 on pci0 pci2: <ACPI PCI bus> on pcib2 pcib3: <PCI-PCI bridge> irq 40 at device 0.0 on pci2 pci3: <PCI bus> on pcib3 pcib4: <PCI-PCI bridge> irq 43 at device 3.0 on pci3 pci4: <PCI bus> on pcib4 re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0xfe900000-0xfe900fff,0xd0900000-0xd0903fff irq 43 at device 0.0 on pci4 re1: Using 1 MSI-X message re1: ASPM disabled re1: Chip rev. 0x2c800000 re1: MAC rev. 0x00100000 miibus1: <MII bus> on re1 rgephy1: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus1 rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re1: Using defaults for TSO: 65518/35/2048 re1: Ethernet address: 00:e0:4c:68:99:9b re1: netmap queues/slots: TX 1/256, RX 1/256 pcib5: <PCI-PCI bridge> irq 43 at device 7.0 on pci3 pci5: <PCI bus> on pcib5 re2: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem 0xfe800000-0xfe800fff,0xd0800000-0xd0803fff irq 43 at device 0.0 on pci5 re2: Using 1 MSI-X message re2: ASPM disabled re2: Chip rev. 0x2c800000 re2: MAC rev. 0x00100000 miibus2: <MII bus> on re2 rgephy2: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus2 rgephy2: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re2: Using defaults for TSO: 65518/35/2048 re2: Ethernet address: 00:e0:4c:68:99:9c re2: netmap queues/slots: TX 1/256, RX 1/256 xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb68000-0xfeb69fff irq 18 at device 16.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xfeb6e000-0xfeb6e3ff irq 19 at device 17.0 on pci0 ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ohci0: <AMD FCH USB Controller> mem 0xfeb6d000-0xfeb6dfff irq 18 at device 18.0 on pci0 usbus1 on ohci0 usbus1: 12Mbps Full Speed USB v1.0 ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb6c000-0xfeb6c0ff irq 17 at device 18.2 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci0 usbus2: 480Mbps High Speed USB v2.0 ohci1: <AMD FCH USB Controller> mem 0xfeb6b000-0xfeb6bfff irq 18 at device 19.0 on pci0 usbus3 on ohci1 usbus3: 12Mbps Full Speed USB v1.0 ehci1: <AMD FCH USB 2.0 controller> mem 0xfeb6a000-0xfeb6a0ff irq 17 at device 19.2 on pci0 usbus4: EHCI version 1.0 usbus4 on ehci1 usbus4: 480Mbps High Speed USB v2.0 hdac1: <AMD Hudson-2 HDA Controller> mem 0xfeb60000-0xfeb63fff irq 16 at device 20.2 on pci0 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 acpi_button0: <Power Button> on acpi0 ns8250: UART FCR is broken ns8250: UART FCR is broken uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) orm0: <ISA Option ROMs> at iomem 0xc0000-0xce7ff,0xce800-0xcf7ff,0xcf800-0xd07ff,0xd0800-0xd17ff pnpid ORM0000 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] hwpstate0: <Cool`n'Quiet 2.0> on cpu0 Timecounter "TSC" frequency 1646911000 Hz quality 1000 Timecounters tick every 1.000 msec hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0 hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0 pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0 pcm1: <ATI R6xx (HDMI)> at nid 5 on hdaa0 hdacc1: <Realtek ALC671 HDA CODEC> at cad 2 on hdac1 hdaa1: <Realtek ALC671 Audio Function Group> at nid 1 on hdacc1 pcm2: <Realtek ALC671 (Analog)> at nid 23 and 24 on hdaa1 pcm3: <Realtek ALC671 (Front Analog)> at nid 20 and 25 on hdaa1 pcm4: <Realtek ALC671 (Rear Analog Line-out)> at nid 33 on hdaa1 Trying to mount root from ufs:/dev/ufsid/5f7488a71a3e644c [rw,noatime]... ugen3.1: <AMD OHCI root HUB> at usbus3 ugen4.1: <AMD EHCI root HUB> at usbus4 ugen1.1: <AMD OHCI root HUB> at usbus1 ugen2.1: <AMD EHCI root HUB> at usbus2 ugen0.1: <AMD XHCI root HUB> at usbus0 uhub1 on usbus4 uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4 sysctl_warn_reuse: can't re-use a leaf (dev.uhub.%parent)! uhub0 on usbus3 uhub0: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 sysctl_warn_reuse: can't re-use a leaf (dev.uhub.%parent)! uhub2 on usbus1 sysctl_warn_reuse: can't re-use a leaf (dev.uhub.%parent)! uhub4 on usbus0 uhub2: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 sysctl_warn_reuse: can't re-use a leaf (dev.uhub.%parent)! uhub3 on usbus2 uhub3: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 uhub4: <AMD XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub0: 4 ports with 4 removable, self powered uhub2: 4 ports with 4 removable, self powered uhub4: 4 ports with 4 removable, self powered Root mount waiting for: CAM usbus2 usbus4 uhub1: 4 ports with 4 removable, self powered uhub3: 4 ports with 4 removable, self powered Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: <Innodisk DEMSR- 16GB mSATA 3ME A S141002> ACS-2 ATA SATA 3.x device ada0: Serial Number 20150708AA1010000026 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 15272MB (31277232 512 byte sectors) Dual Console: Serial Primary, Video Secondary Setting hostuuid: fd799591-702f-e511-ab9a-901b0e5f43da. Setting hostid: 0x4259b132. Starting file system checks: /dev/ufsid/5f7488a71a3e644c: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufsid/5f7488a71a3e644c: clean, 2830788 free (7308 frags, 352935 blocks, 0.2% fragmentation) Mounting local filesystems:. Generating MFS /var partition pid 83 (bsdcpio), uid 0 inumber 41 on /tmp: filesystem full /tmp: write failed, filesystem is full cpio: Write failed: No space left on device Generating MFS /etc partition Autoloading module: intpm intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0 smbus0: <System Management Bus> on intsmb0 Local package initialization: (skipping /usr/local/etc/rc.d/named.sh, not executable). re0: link state changed to UP re1: link state changed to UP re2: link state changed to UP /etc/rc: WARNING: $hostname is not set -- see rc.conf(5). Setting up harvesting: [CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP Starting Network: lo0 re0 re1 re2 enc0. lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> re0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 90:1b:0e:5f:43:da media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> re1: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 00:e0:4c:68:99:9b media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> re2: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 00:e0:4c:68:99:9c media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> enc0: flags=0 metric 0 mtu 1536 options=0 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/lib/engines /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE 32-bit compatibility ldconfig path: Starting devd. Starting Network: re0. re0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 90:1b:0e:5f:43:da media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Starting Network: re1. re1: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 00:e0:4c:68:99:9b media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Starting Network: re2. re2: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 00:e0:4c:68:99:9c media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Starting Network: enc0. enc0: flags=0 metric 0 mtu 1536 options=0 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> . Creating and/or trimming log filespid 1236 (newsyslog), uid 0 inumber 2332 on /var: filesystem full /var: write failed, filesystem is full newsyslog: log_trim: fclose: No space left on device . Clearing /tmp (X related). Updating motd:install: /var/run/motd: No space left on device . Updating /var/run/os-release install: /var/run/os-release: No space left on device done. Starting syslogd. protect: option requires an argument -- p usage: protect [-i] command protect [-cdi] -g pgrp | -p pid No core dumps found. /etc/rc: WARNING: $timed_enable is not set properly - see rc.conf(5). pututxline: No space left on device assword:late filesystems:. /etc/rc: WARNING: $scponlyc_enable is not set properly - see rc.conf(5). /etc/rc: WARNING: $sshguard_enable is not set properly - see rc.conf(5). echo: write error on stdout eval: write error on stdout Starting cron. Starting background file system checks in 60 seconds. Mon Nov 27 15:05 FreeBSD/amd64 (Amnesiac) (ttyu0) login: -
Filesystems right after boot
root@:~ # df -kh Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/5f7488a71a3e644c 14G 2.8G 9.7G 22% / devfs 1.0K 0B 1.0K 0% /dev /dev/md0 31M 31M -624K 102% /var /dev/md1 19M 8.1M 10M 44% /etc /dev/md2 31M 2.7M 25M 10% /usr/local/etc -
Disk usage right after boot
root@:/ # du -skh /* 8.0K /COPYRIGHT 1.4M /bin 414M /boot 4.0K /boot.config 2.1M /cf 0B /conf 20K /conf.default 3.5K /dev 4.0K /entropy 8.1M /etc 4.0K /home 22M /lib 164K /libexec 4.0K /media 4.0K /mnt 4.0K /net 4.0K /proc 27M /rescue 36K /root 4.7M /sbin 0B /sys 80K /tmp 1.7G /usr 31M /var root@:/ # du -skh /var/* 4.0K /var/account 12K /var/at 12K /var/audit 4.0K /var/authpf 4.0K /var/backups 22M /var/cache 4.0K /var/crash 8.0K /var/cron 8.8M /var/db 4.0K /var/empty 4.0K /var/games 4.0K /var/heimdal 40K /var/log 4.0K /var/mail 4.0K /var/msgs 4.0K /var/preserve 32K /var/run 4.0K /var/rwho 24K /var/spool 8.0K /var/tmp 8.0K /var/unbound 4.0K /var/yp -
@redpola said in Out of disk during upgrade to 23.09, system boots (badly) but I can ssh in...:
/dev/md0 31M 31M -624K 102% /var
/var of 32 MB seems incredibly small. You should be able to delete files in /var/log. Some packages will write into /var as well.
I would plan to reinstall just to fix that...
-
I'd also consider to switch to ZFS on this occasion.
-
@SteveITS Thanks for the response. Unfortunately the partitioning was not my decision- I would have accepted the defaults offered to me at the time.
-
@redpola It won't help with the upgrade but one can use a RAM disk for /var and /tmp. However, pfSense will copy data to disk during boots. There is a "Log Directory" time in hours, but even if left blank I suspect it will copy at restart. And of course for an unexpected boot you'd lose all logs at boot. And note their note:
"The size of the /var RAM disk, in MiB. The default value is 60, but should be set much higher, especially if packages will be used. 512-1024 is a better starting point, depending on the available firewall RAM and kernel memory."
...hence why 32 MB seems small.
-
I just wanted to finish up this story for anybody who found it because they are in the same predicament.
I never found a way to rescue the half-upgraded installation, but I did manage to dump config.xml over my serial console.
Ultimately I downloaded the latest usb stick image with serial console, booted from it, and did a reinstall (over my serial console).
The install was fast, upgraded the whole thing to ZFS, and - much to my surprise - found and used my existing config.xml
I instructed the machine to reboot and again, was extremely surprised to find that my relatively complex configuration was back up and working with no more intervention. I logged into the web interface and found a message saying packages were being automatically installed, and not to change anything in the GUI and that was it.
I'm very impressed at what was a particularly easy and intuitive experience.
Thanks for anyone who offered help!