2.8.1 to 2.9 : UI and WAN down post upgrade
-
Post upgrade issue
Main WAN is down (FTTH)
Backup WAN is working (hence I'm able to post) (3G access point on secondary ethernet)
Ping to router LAN interface OK
Web UI: Down (The web server encountered an error processing this request.) crash_reporter.php shows the same error
SSH to router: downSo I have basically no information whatsoever.
I've already soft shutdown the router using the power button, and restarted it: same behavior -
ok, so after hooking up a KVM and rebooting I managed to restart PHP-FHM from the menu and I have the UI back and disabling/enabling my main WAN interface brought it back online.
How can I check what went wrong and fix it to avoid this issue (UI) next time the router restarts ?
Logs shows a bunch of these:
Aug 24 12:50:14 check_reload_status 498 Could not connect to /var/run/php-fpm.socket(I also had to reinstall wireguard as it wouldn't start)
-
I also have a couple of these:
Aug 24 13:04:49 php-fpm 46284 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787569489] unbound[43638:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787569489] unbound[43638:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787569489] unbound[43638:0] error: bind: address already in use [1787569489] unbound[43638:0] fatal error: could not open ports' -
Do you see one of the interfaces flapping at boot?
The errors you have posted there all look like symptoms not causes.
-
@stephenw10 all my interfaces are UP at boot, never been an issue.
Often, my IPV4 gateway shows offline at boot (even though the If gets its IP), but disabling/enabling the WAN interface fixes it. (my IPV6 WAN, 3G WAN are always fine)
My main issue is having to hook up a KVM to restart PHP-FPM from the menu after a reboot
-
Probably need to see the full boot log to see what's failing there then.
-
I have about 5 system.log full of this (since they rotate every 500Kb)
pfSense check_reload_status[498]: Could not connect to /var/run/php-fpm.socketboot log
-
Hmm, is that just the boot log rather then the system log covering the bootup?
Unclear which though it does show some info after the bootup completes.
But what was the state after the 1st boot? It looks like you logged in locally and rebooted after that.
What are those interfaces physically connected to. At least igc2 and igc0 seem to flap a lot at boot.
I suspect you are hitting a race condition that eventually blocks all the available php threads. Is that CPU running at it's expected speed? The N100/N200 is somewhat notorious for running slow/hot in some cheap chinese boxes that ship with generic bioses. Obviously a race condition shouldn't happen either way.
-
igc0 = FTTH WAN (primary WAN)
igc1 = LAN
igc2 = LTE/3G access point (failback WAN)after first boot, I couldn't access UI, ssh was disabled (or failed to start).
I had to power off/power on the unit, connect a KVM and restart PHP-FPM using the menu, then log in UI and manually start SSH server(I also had to disable/enable igc0/WAN interface as the gateway was show offline. but this particular bit has been an issue for a long time, not a big deal.)
It is a N100 / Topton with an unlocked BIOS ( https://forum.netgate.com/post/1228501)
Boot log:
TSC: P-state invariant, performance statistics VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr IA32_ARCH_CAPS=0x15c0fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> Structured Extended Features4=0x810 Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD> Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B> Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA> AMD Features2=0x121<LAHF,ABM,Prefetch> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Origin="GenuineIntel" Id=0xb06e0 Family=0x6 Model=0xbe Stepping=0 CPU: Intel(R) N100 (806.40-MHz K8-class CPU) nda0: 122104MB (250069680 512 byte sectors) nda0: nvme version 1.3 nda0: Serial Number AA000000000000000316 nda0: <NVME SSD 128GB APF1M3R1 AA000000000000000316> nda0 at nvme0 bus 0 scbus1 target 0 lun 1 Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM hidbus1: <HID bus> on usbhid1 usbhid1: <vendor 0x13ba Barcode Reader, class 0/0, rev 1.10/0.01, addr 1> on usbus1 usbhid1 on uhub1 kbd1 at hkbd0 hkbd0: <vendor 0x13ba Barcode Reader Keyboard> on hidbus0 hidbus0: <HID bus> on usbhid0 usbhid0: <vendor 0x13ba Barcode Reader, class 0/0, rev 1.10/0.01, addr 1> on usbus1 usbhid0 on uhub1 ugen1.2: <vendor 0x13ba Barcode Reader> at usbus1 Root mount waiting for: usbus1 CAM uhub1: 16 ports with 16 removable, self powered uhub0: 3 ports with 3 removable, self powered Trying to mount root from zfs:pfSense/ROOT/default []... nvme_sim0: <nvme cam> on nvme0 nvme0: Allocated 127MB host memory buffer uhub1: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 uhub1 on usbus1 ZFS storage pool version: features support (5000) ZFS filesystem version: 5 uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub0 on usbus0 ugen1.1: <Intel XHCI root HUB> at usbus1 ugen0.1: <Intel XHCI root HUB> at usbus0 Timecounters tick every 1.000 msec Timecounter "TSC" frequency 806401044 Hz quality 1000 cpufreq3: <CPU frequency control> on cpu3 hwpstate_intel3: <Intel Speed Shift> on cpu3 cpufreq2: <CPU frequency control> on cpu2 hwpstate_intel2: <Intel Speed Shift> on cpu2 cpufreq1: <CPU frequency control> on cpu1 hwpstate_intel1: <Intel Speed Shift> on cpu1 cpufreq0: <CPU frequency control> on cpu0 hwpstate_intel0: <Intel Speed Shift> on cpu0 atrtc0: Can't map interrupt. atrtc0: registered as a time-of-day clock, resolution 1.000000s atrtc0: Warning: Couldn't map I/O. atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0 acpi_syscontainer1: <System Container> on acpi0 acpi_syscontainer0: <System Container> on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart: ns8250: UART FCR is broken (0x1) uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart: ns8250: UART FCR is broken (0x1) acpi_tz0: <Thermal Zone> on acpi0 acpi_button1: <Power Button> on acpi0 acpi_spmc0: DSM Intel, revision 0: Supported functions: 0x7e<DEVICE_CONSTRAINTS,CRASH_DUMP_DEVICE,DISPLAY_OFF,DISPLAY_ON,LPI_ENTRY,LPI_EXIT> acpi_spmc0: DSMs supported: <Intel> acpi_spmc0: <System Power Management Controller> on acpi0 cpu0: <ACPI CPU> on acpi0 acpi_button0: <Sleep Button> on acpi0 pci0: <serial bus> at device 31.5 (no driver attached) hdac0: <Intel Alder Lake-N HDA Controller> mem 0x6001120000-0x6001123fff,0x6001000000-0x60010fffff at device 31.3 on pci0 isa0: <ISA bus> on isab0 isab0: <PCI-ISA bridge> at device 31.0 on pci0 nvme0: <Generic NVMe Device> mem 0x81010000-0x81013fff at device 0.0 on pci5 pci5: <ACPI PCI bus> on pcib5 pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci0 igc3: netmap queues/slots: TX 4/1024, RX 4/1024 igc3: Ethernet address: 00:d0:b4:02:14:71 igc3: Using MSI-X interrupts with 5 vectors igc3: Using 4 RX queues 4 TX queues igc3: Using 1024 TX descriptors and 1024 RX descriptors igc3: EEPROM V2.13-0 eTrack 0x80000284 igc3: <Intel(R) Ethernet Controller I226-V> mem 0x80500000-0x805fffff,0x80600000-0x80603fff at device 0.0 on pci4 pci4: <ACPI PCI bus> on pcib4 pcib4: <ACPI PCI-PCI bridge> at device 28.6 on pci0 igc2: netmap queues/slots: TX 4/1024, RX 4/1024 igc2: Ethernet address: 00:d0:b4:02:14:70 igc2: Using MSI-X interrupts with 5 vectors igc2: Using 4 RX queues 4 TX queues igc2: Using 1024 TX descriptors and 1024 RX descriptors igc2: EEPROM V2.13-0 eTrack 0x80000284 igc2: <Intel(R) Ethernet Controller I226-V> mem 0x80800000-0x808fffff,0x80900000-0x80903fff at device 0.0 on pci3 pci3: <ACPI PCI bus> on pcib3 pcib3: <ACPI PCI-PCI bridge> at device 28.2 on pci0 igc1: netmap queues/slots: TX 4/1024, RX 4/1024 igc1: Ethernet address: 00:d0:b4:02:14:6f igc1: Using MSI-X interrupts with 5 vectors igc1: Using 4 RX queues 4 TX queues igc1: Using 1024 TX descriptors and 1024 RX descriptors igc1: EEPROM V2.13-0 eTrack 0x80000284 igc1: <Intel(R) Ethernet Controller I226-V> mem 0x80b00000-0x80bfffff,0x80c00000-0x80c03fff at device 0.0 on pci2 pci2: <ACPI PCI bus> on pcib2 pcib2: <ACPI PCI-PCI bridge> at device 28.1 on pci0 igc0: netmap queues/slots: TX 4/1024, RX 4/1024 igc0: Ethernet address: 00:d0:b4:02:14:6e igc0: Using MSI-X interrupts with 5 vectors igc0: Using 4 RX queues 4 TX queues igc0: Using 1024 TX descriptors and 1024 RX descriptors igc0: EEPROM V2.13-0 eTrack 0x80000284 igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80e00000-0x80efffff,0x80f00000-0x80f03fff at device 0.0 on pci1 pci1: <ACPI PCI bus> on pcib1 pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahci0: AHCI v1.31 with 1 6Gbps ports, Port Multiplier not supported ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0x81100000-0x81101fff,0x81103000-0x811030ff,0x81102000-0x811027ff at device 23.0 on pci0 pci0: <simple comms> at device 22.0 (no driver attached) pci0: <memory, RAM> at device 20.2 (no driver attached) usbus1: 5.0Gbps Super Speed USB v3.0 usbus1 on xhci1 xhci1: xECP capabilities <PROTO,PROTO,VEND(c0),LEGACY,VEND(c6),VEND(c7),VEND(c2),DEBUG,VEND(c3),VEND(c4),VEND(ce),VEND(c8),VEND(c9),VEND(ca),VEND(cb),VEND(cc),VEND(cd)> xhci1: 32 bytes context size, 64-bit DMA xhci1: <Intel Alder Lake USB 3.2 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0 usbus0: 5.0Gbps Super Speed USB v3.0 usbus0 on xhci0 xhci0: xECP capabilities <PROTO,PROTO,VEND(c0),LEGACY,VEND(c6),VEND(c7),VEND(c2),DEBUG,VEND(c3),VEND(d1),VEND(ce),VEND(c8),VEND(c9),VEND(ca),VEND(cc),VEND(cd),VEND(d2),VEND(cf),VEND(d3)> xhci0: 32 bytes context size, 64-bit DMA xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001110000-0x600111ffff at device 13.0 on pci0 vgapci0: Boot video device vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff at device 2.0 on pci0 pci0: <ACPI PCI bus> on pcib0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "i8254" frequency 1193182 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 atrtc1: registered as a time-of-day clock, resolution 1.000000s atrtc1: Warning: Couldn't map I/O. atrtc1: <AT realtime clock> on acpi0 Event timer "HPET4" frequency 19200000 Hz quality 440 Event timer "HPET3" frequency 19200000 Hz quality 440 Event timer "HPET2" frequency 19200000 Hz quality 440 Event timer "HPET1" frequency 19200000 Hz quality 440 Event timer "HPET" frequency 19200000 Hz quality 550 Timecounter "HPET" frequency 19200000 Hz quality 950 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 acpi0: Power Button (fixed) acpi0: <ALASKA A M I > smbios0: Entry point: v3 (64-bit), Version: 3.5 smbios0: <System Management BIOS> at iomem 0x75cc4000-0x75cc4017 netgate0: version: 0.2 netgate0: <unknown hardware> efirtc0: registered as a time-of-day clock, resolution 1.000000s efirtc0: <EFI Realtime Clock> kbd0 at kbdmux0 wlan: mac acl policy registered random: entropy device external interface module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807c4eb0, 0) error 1 iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff807c4e00, 0) error 1 iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff807c4d50, 0) error 1 iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff807a5290, 0) error 1 ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff807a51e0, 0) error 1 ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807a5130, 0) error 1 ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. TCP_ratelimit: Is now initialized Launching APs: 2 3 1 ioapic0 <Version 2.0> irqs 0-119 random: unblocking device. random: fast provider: "Intel Secure Key Seed" random: registering fast source Intel Secure Key Seed FreeBSD/SMP: 1 package(s) x 4 core(s) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs WARNING: L3 data cache covers more APIC IDs than a package (7 > 3) ACPI APIC Table: <ALASKA A M I > Event timer "LAPIC" quality 600 avail memory = 8023773184 (7652 MB) real memory = 8589934592 (8192 MB) TSC: P-state invariant, performance statistics VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr IA32_ARCH_CAPS=0x180fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> Structured Extended Features4=0x810 Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD> Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B> Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA> AMD Features2=0x121<LAHF,ABM,Prefetch> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Origin="GenuineIntel" Id=0xb06e0 Family=0x6 Model=0xbe Stepping=0 CPU: Intel(R) N100 (806.40-MHz K8-class CPU) sysctl_register_oid: can't re-use a leaf (vfs.zfs.metaslab.condense_pct)! VT(efifb): resolution 800x600 FreeBSD clang version 21.1.8 (https://github.com/llvm/llvm-project.git llvmorg-21.1.8-0-g2078da43e25a) root@pfsense-build-release-amd64-2.eng.atx.netgate.com:/var/jenkins/workspace/pfSense-CE-snapshots-2_9_0-main/obj/amd64/XcuObCIF/var/jenkins/workspace/pfSense-CE-snapshots-2_9_0-main/sources/FreeBSD-src-RELENG_2_9_0/amd64.amd64/sys/pfSense amd64 FreeBSD 16.0-CURRENT #12 RELENG_2_9_0-n256132-d8e3138ecf52: Mon Aug 17 18:50:13 UTC 2026 FreeBSD is a registered trademark of The FreeBSD Foundation. The Regents of the University of California. All rights reserved. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Copyright (c) 1992-2026 The FreeBSD Project. -
Yeah the system log covering the boot is actually more useful here.
Hmm, I assume the devices connected to those interfaces are not rebooting at the same time? Or for some other reason bouncing the interface?
-
@stephenw10 correct, nothing else was restarted
php-fpm.log
[24-Aug-2026 12:00:02] ERROR: No pool defined. at least one pool section must be specified in config file [24-Aug-2026 12:00:02] ERROR: failed to post process the configuration [24-Aug-2026 12:00:02] ERROR: FPM initialization failed [24-Aug-2026 12:12:20] ERROR: No pool defined. at least one pool section must be specified in config file [24-Aug-2026 12:12:20] ERROR: failed to post process the configuration [24-Aug-2026 12:12:20] ERROR: FPM initialization failed [24-Aug-2026 12:47:41] ERROR: No pool defined. at least one pool section must be specified in config file [24-Aug-2026 12:47:41] ERROR: failed to post process the configuration [24-Aug-2026 12:47:41] ERROR: FPM initialization failed -
Hmm interesting. Not sure I've seen that error before.
-
Hello, @stephenw10 @Le_zOU I have the same failure on 2.9.0 after an upgrade. I think I found a lead. Same symptoms as the original post: web GUI returns the 50x error page, SSH never starts, packet filtering keeps working. Roughly 1 boot out of 3 on my box.
When it fails, /usr/local/lib/php-fpm.conf reads back as 650 bytes of NUL. Correct size, zero content, so no [nginx] section. Hence No pool defined and php-fpm never starts. nginx stays up and serves the error page.
# wc -lc /usr/local/lib/php-fpm.conf 0 650# od -c /usr/local/lib/php-fpm.conf | head -3 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001212/usr/local/etc/php.ini is written by the same rc.php_ini_setup run, in the same second, and it is completely intact (45 lines, 1293 bytes). Only php-fpm.conf is null. mtime on both files is identical and matches the No pool defined timestamp to the second.
Storage looks clean. zpool status -v: ONLINE, 0/0/0 READ/WRITE/CKSUM, no known data errors, scrub repaired 0B. feature@block_cloning is disabled. SMART PASSED on the M.2 SSD. sync=standard, compression=lz4.
I think generator is not the problem. I ran rc.php_ini_setup 20 times in a loop on a healthy system: 20/20 produced valid file.
Once broken, it stays broken for the rest of the boot. I hit option 16 twice in the same boot and both attempts failed identically, even though that path regenerates the file first.
Setup: pfSense CE 2.9.0-RELEASE amd64 20260828-1753, ZFS root on a single-disk pool, 16 GB RAM, N100 cpu.
Privacy Policy · Cookie Policy