Router Locking Up (maybe due to excessive lan traffic?)
-
EFI systbl not available firmware: 'iwm3160fw' version 0: 918268 bytes loaded at 0xffffffff81b196f6 module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1 iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. firmware: 'rt2860fw' version 0: 8192 bytes loaded at 0xffffffff82acc5ce firmware: 'rt2661fw' version 0: 8192 bytes loaded at 0xffffffff82aca5ce firmware: 'rt2561sfw' version 0: 8192 bytes loaded at 0xffffffff82ac85ce firmware: 'rt2561fw' version 0: 8192 bytes loaded at 0xffffffff82ac65ce module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1 iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1 iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1 ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1 ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1 ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. firmware: 'runfw' version 0: 8192 bytes loaded at 0xffffffff82b18392 firmware: 'rsu-rtl8712fw' version 120: 122328 bytes loaded at 0xffffffff82afa5ba firmware: 't6fw' version 0: 730624 bytes loaded at 0xffffffff819491f2 firmware: 't6fw_cfg_uwire' version 0: 22693 bytes loaded at 0xffffffff8194394d firmware: 't6fw_cfg' version 0: 5960 bytes loaded at 0xffffffff81942205 firmware: 't5fw' version 0: 678912 bytes loaded at 0xffffffff8189c605 firmware: 't5fw_cfg_uwire' version 0: 21859 bytes loaded at 0xffffffff818970a2 firmware: 't5fw_cfg' version 0: 5902 bytes loaded at 0xffffffff81895994 wlan: <802.11 Link Layer> firmware: 't4fw' version 0: 570880 bytes loaded at 0xffffffff8180a394 firmware: 't4fw_cfg_uwire' version 0: 21433 bytes loaded at 0xffffffff81804fdb firmware: 't4fw_cfg' version 0: 4523 bytes loaded at 0xffffffff81803e30 firmware: 'iwn6050fw' version 0: 469780 bytes loaded at 0xffffffff82a3bef6 firmware: 'iwn6000g2bfw' version 0: 679436 bytes loaded at 0xffffffff829960ea firmware: 'iwn6000g2afw' version 0: 677296 bytes loaded at 0xffffffff828f0b3a firmware: 'iwn6000fw' version 0: 454608 bytes loaded at 0xffffffff82881b6a firmware: 'iwn5150fw' version 0: 337400 bytes loaded at 0xffffffff8282f572 firmware: 'iwn5000fw' version 0: 340696 bytes loaded at 0xffffffff827dc29a wlan: mac acl policy registered firmware: 'mxge_rss_ethp_z8e' version 544468: 152837 bytes loaded at 0xffffffff82cafd20 firmware: 'mxge_rss_eth_z8e' version 534724: 151912 bytes loaded at 0xffffffff82c8ab00 firmware: 'mxge_ethp_z8e' version 387604: 121317 bytes loaded at 0xffffffff82c6d060 firmware: 'mxge_eth_z8e' version 377284: 120629 bytes loaded at 0xffffffff82c4f870 firmware: 'mwlboot' version 0: 2280 bytes loaded at 0xffffffff82ac5ce6 firmware: 'mw88W8363fw' version 0: 94940 bytes loaded at 0xffffffff82aaea0a feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=2 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25 snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024] firmware: 'iwn4965fw' version 0: 187972 bytes loaded at 0xffffffff827ae456 firmware: 'iwn2030fw' version 0: 707392 bytes loaded at 0xffffffff82701916 firmware: 'iwn2000fw' version 0: 695876 bytes loaded at 0xffffffff82657ad2 firmware: 'iwn135fw' version 0: 701228 bytes loaded at 0xffffffff825ac7a6 firmware: 'iwn105fw' version 0: 689680 bytes loaded at 0xffffffff82504196 firmware: 'iwn100fw' version 0: 337572 bytes loaded at 0xffffffff824b1af2 firmware: 'iwn1000fw' version 0: 337520 bytes loaded at 0xffffffff8245f482 firmware: 'iwm8265fw' version 0: 1811984 bytes loaded at 0xffffffff822a4e72 firmware: 'rtwn-rtl8821aufw' version 111: 27804 bytes loaded at 0xffffffff82af391e firmware: 'rtwn-rtl8812aufw' version 111: 32060 bytes loaded at 0xffffffff82aebbe2 firmware: 'rtwn-rtl8192eufw' version 111: 31818 bytes loaded at 0xffffffff82ae3f98 firmware: 'rtwn-rtl8192cfwU' version 111: 16126 bytes loaded at 0xffffffff82ae009a firmware: 'rtwn-rtl8192cfwT' version 111: 16126 bytes loaded at 0xffffffff82adc19c firmware: 'rtwn-rtl8192cfwE_B' version 111: 16332 bytes loaded at 0xffffffff82ad81d0 firmware: 'wpifw' version 153229: 150100 bytes loaded at 0xffffffff82b1a392 firmware: 'rtwn-rtl8192cfwE' version 111: 14818 bytes loaded at 0xffffffff82ad47ee firmware: 'rtwn-rtl8188eufw' version 111: 13904 bytes loaded at 0xffffffff82ad119e firmware: 'rtwn-rtl8188eefw' version 111: 11216 bytes loaded at 0xffffffff82ace5ce firmware: 'iwm8000Cfw' version 0: 2351636 bytes loaded at 0xffffffff82066c5e firmware: 'iwm7265Dfw' version 0: 1383604 bytes loaded at 0xffffffff81f14faa firmware: 'iwm7265fw' version 0: 1180412 bytes loaded at 0xffffffff81df4cae firmware: 'iwm7260fw' version 0: 1049340 bytes loaded at 0xffffffff81cf49b2 firmware: 'iwm3168fw' version 0: 1028032 bytes loaded at 0xffffffff81bf99f2 random: entropy device external interface TCP_ratelimit: Is now initialized SMP: passed TSC synchronization test elvt0: 0x00010000 elvt1: 0x00010000 elvt2: 0x00010000 elvt3: 0x00010000 AMD ext features: 0x00040007 timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff ID: 0x01000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff cpu1 AP: SMP: AP CPU #1 Launched! elvt0: 0x00010000 elvt1: 0x00010000 elvt2: 0x00010000 elvt3: 0x00010000 AMD ext features: 0x00040007 timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff ID: 0x03000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff cpu3 AP: SMP: AP CPU #3 Launched! elvt0: 0x00010000 elvt1: 0x00010000 elvt2: 0x00010000 elvt3: 0x00010000 AMD ext features: 0x00040007 timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff ID: 0x02000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff cpu2 AP: SMP: AP CPU #2 Launched! elvt0: 0x00010000 elvt1: 0x000000f2 elvt2: 0x00010000 elvt3: 0x00010000 AMD ext features: 0x00040007 timer: 0x000300ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff ID: 0x00000000 VER: 0x80050010 LDR: 0x00000000 DFR: 0xffffffff cpu0 BSP: ioapic1 <Version 2.1> irqs 24-55 ioapic0 <Version 2.1> irqs 0-23 lapic: LINT1 polarity: high lapic: LINT1 trigger: edge lapic: Routing NMI -> LINT1 ioapic0: intpin 9 polarity: low ioapic0: intpin 9 trigger: level MADT: Interrupt override: source 9, irq 9 ioapic0: Routing IRQ 0 -> intpin 2 MADT: Interrupt override: source 0, irq 2 ioapic1: ver 0x21 maxredir 0x1f ioapic1: MADT APIC ID 5 != hw id 0 MADT: Found IO APIC ID 5, Interrupt 24 at 0xfec20000 ioapic0: Routing external 8259A's -> intpin 0 ioapic0: ver 0x21 maxredir 0x17 MADT: Found IO APIC ID 4, Interrupt 0 at 0xfec00000 ACPI: HPET 0x00000000DFFBEB10 000038 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: SSDT 0x00000000DFFBE340 0007C8 (v01 AMD AGESA 00000001 AMD 00000001) ACPI: SSDT 0x00000000DFFB9A90 0048A6 (v02 AMD AGESA 00000002 MSFT 04000000) ACPI: HEST 0x00000000DFFB98C0 0001D0 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: APIC 0x00000000DFFB9840 00007E (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: SSDT 0x00000000DFFB97F0 000045 (v02 CORE COREBOOT 0000002A CORE 0000002A) ACPI: FACS 0x00000000DFFB8210 000040 ACPI: DSDT 0x00000000DFFB8250 001496 (v02 AMD COREBOOT 00010001 INTL 20140114) ACPI: FACP 0x00000000DFFB96F0 0000F4 (v04 CORE COREBOOT 00000000 CORE 00000000) ACPI: XSDT 0x00000000DFFB80E0 00005C (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: RSDP 0x00000000000F4840 000024 (v02 CORE ) ULE: setup cpu 3 ULE: setup cpu 2 ULE: setup cpu 1 ULE: setup cpu 0 hostuuid: using 00000000-0000-0000-0000-000000000000 VIMAGE (virtualized network stack) enabled random: unblocking device. random: no platform bootloader entropy random: read 4096 bytes from preloaded cache x86bios: ROM 0x0a0000-0x0fefff at 0xfffff800000a0000 x86bios: EBDA 0x09f000-0x09ffff at 0xfffff8000009f000 x86bios: SSEG 0x09e000-0x09efff at 0xfffffe0008531000 x86bios: IVT 0x000000-0x0004ff at 0xfffff80000000000 Pentium Pro MTRR support enabled APIC: CPU 3 has ACPI ID 3 APIC: CPU 2 has ACPI ID 2 APIC: CPU 1 has ACPI ID 1 APIC: CPU 0 has ACPI ID 0 CPU3 (AP): APIC ID: 3 Core HW ID = 3 CPU2 (AP): APIC ID: 2 Core HW ID = 2 CPU1 (AP): APIC ID: 1 Core HW ID = 1 CPU0 (BSP): APIC ID: 0 Core HW ID = 0 Package HW ID = 0 FreeBSD/SMP: 1 package(s) x 4 core(s) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs INTR: Adding local APIC 3 as a target INTR: Adding local APIC 2 as a target INTR: Adding local APIC 1 as a target AP boot address 0x9e000 Core ID shift: 0 L1 cache ID shift: 0 L2 cache ID shift: 2 Package ID shift: 3 ACPI APIC Table: <CORE COREBOOT> LAPIC: ipi_wait() us multiplier 5 (r 176182 tsc 998139400) Event timer "LAPIC" quality 100 SMP: Added CPU 3 (AP) MADT: Found CPU APIC ID 3 ACPI ID 3: enabled SMP: Added CPU 2 (AP) MADT: Found CPU APIC ID 2 ACPI ID 2: enabled SMP: Added CPU 1 (AP) MADT: Found CPU APIC ID 1 ACPI ID 1: enabled SMP: Added CPU 0 (AP) MADT: Found CPU APIC ID 0 ACPI ID 0: enabled avail memory = 4071268352 (3882 MB) 0x0000000100000000 - 0x000000011efe7fff, 519995392 bytes (126952 pages) 0x00000000dfe00000 - 0x00000000dff8cfff, 1626112 bytes (397 pages) 0x0000000004002000 - 0x00000000d87a5fff, 3564781568 bytes (870308 pages) 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages) 0x0000000000010000 - 0x000000000009efff, 585728 bytes (143 pages) Physical memory chunk(s): real memory = 4815060992 (4592 MB) L2 unified cache: 2048 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative L2 4KB instruction TLB: 512 entries, 4-way associative L2 4KB data TLB: 512 entries, 4-way associative L2 2MB instruction TLB: 0 entries, 2-way associative L2 2MB data TLB: 256 entries, 2-way associative L1 instruction cache: 32 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L1 data cache: 32 kbytes, 64 bytes/line, 1 lines/tag, 8-way associative L1 4KB instruction TLB: 32 entries, fully associative L1 4KB data TLB: 40 entries, fully associative L1 2MB instruction TLB: 8 entries, fully associative L1 2MB data TLB: 8 entries, fully associative TSC: P-state invariant, performance statistics Revision=1, ASIDs=8 SVM: Features=0x1cdf<NP,LbrVirt,SVML,NRIPS,TscRateMsr,FlushByAsid,DecodeAssist,PauseFilter,EncryptedMcodePatch,PauseFilterThreshold> XSAVE Features=0x1<XSAVEOPT> Structured Extended Features=0x8<BMI1> AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 CPU: AMD GX-412TC SOC (998.14-MHz K8-class CPU) Early TSC frequency 998139400Hz calibrated from 8254 PIT Preloaded TSLOG data "TSLOG" at 0xffffffff83cd9b80. Preloaded boot_entropy_cache "/boot/entropy" at 0xffffffff83cd9b28. Preloaded elf obj module "/boot/kernel/zfs.ko" at 0xffffffff83cd92c0. Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff83cd0000. VT(vga): resolution 640x480 pmap: large map 8 PML4 slots (4096 GB) PPIM 0: PA=0xa0000, VA=0xffffffff84410000, size=0x10000, mode=0 FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64 FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023 FreeBSD is a registered trademark of The FreeBSD Foundation. The Regents of the University of California. All rights reserved. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Copyright (c) 1992-2023 The FreeBSD Project.
-
Yeah that all looks normal. I wouldn't expect it to show anything until stops responding. If it's a driver/hardware issue it might show additional output at that point.
-
@stephenw10 I thought I nipped this issue.. running will for several days until just now.
I have DNS redirection configured according to this recipe:
https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html
except I noticed I had set Destination: !This Firewall instead of LAN Address. I'm not sure the difference, but I changed it to LAN Address several days ago.Just now, the network failed again. GUI via browser reported 504 Gateway Timeout. I was able to RDP to my laptop which is connected to the pfSense console via serial. I pinged:
8.8.8.8 - OK
9.9.9.9 - OK
1.1.1.1 - 100% packet lose
10.0.11.11 - OKVia Console menu, I tried restarting PHP-FPM, and I was then able to open the GUI in the browser. Only issue I saw was the primary gateway offline. Via console, I tried playback restartallwan... waited probably about 5 mins but no response from the console. Went back to the GUI & rebooted. The router didn't come up. Had to power cycle.
Excerpt from the System log shows:
Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
For context...
Mar 13 21:47:07 router kernel: ---<<BOOT>>--- Mar 13 21:47:07 router kernel: Copyright (c) 1992-2023 The FreeBSD Project. Mar 13 21:47:07 router kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Mar 13 21:47:07 router kernel: The Regents of the University of California. All rights reserved. Mar 13 21:47:07 router kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Mar 13 21:47:07 router kernel: FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023 Mar 13 21:47:07 router kernel: root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64 Mar 13 21:47:07 router kernel: FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152) Mar 13 21:47:07 router kernel: VT(vga): resolution 640x480 Mar 13 21:47:07 router kernel: CPU: AMD GX-412TC SOC (998.28-MHz K8-class CPU) Mar 13 21:47:07 router kernel: Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1 Mar 13 21:47:07 router kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Mar 13 21:47:07 router kernel: Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> Mar 13 21:47:07 router kernel: AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> Mar 13 21:47:07 router kernel: AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I> Mar 13 21:47:07 router kernel: Structured Extended Features=0x8<BMI1> Mar 13 21:47:07 router kernel: XSAVE Features=0x1<XSAVEOPT> Mar 13 21:47:07 router kernel: SVM: NP,NRIP,AFlush,DAssist,NAsids=8 Mar 13 21:47:07 router kernel: TSC: P-state invariant, performance statistics Mar 13 21:47:07 router kernel: real memory = 4815060992 (4592 MB) Mar 13 21:47:07 router kernel: avail memory = 4071268352 (3882 MB) Mar 13 21:47:07 router kernel: Event timer "LAPIC" quality 100 Mar 13 21:47:07 router kernel: ACPI APIC Table: <CORE COREBOOT> Mar 13 21:47:07 router kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Mar 13 21:47:07 router kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) Mar 13 21:47:07 router kernel: random: unblocking device. Mar 13 21:47:07 router kernel: ioapic1: MADT APIC ID 5 != hw id 0 Mar 13 21:47:07 router kernel: ioapic0 <Version 2.1> irqs 0-23 Mar 13 21:47:07 router kernel: ioapic1 <Version 2.1> irqs 24-55 Mar 13 21:47:07 router kernel: Launching APs: 2 1 3 Mar 13 21:47:07 router kernel: TCP_ratelimit: Is now initialized Mar 13 21:47:07 router kernel: random: entropy device external interface Mar 13 21:47:07 router kernel: wlan: mac acl policy registered Mar 13 21:47:07 router kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Mar 13 21:47:07 router kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1 Mar 13 21:47:07 router kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Mar 13 21:47:07 router kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1 Mar 13 21:47:07 router kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. Mar 13 21:47:07 router kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1 Mar 13 21:47:07 router kernel: iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Mar 13 21:47:07 router kernel: iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1 Mar 13 21:47:07 router kernel: iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Mar 13 21:47:07 router kernel: iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1 Mar 13 21:47:07 router kernel: iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. Mar 13 21:47:07 router kernel: iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. Mar 13 21:47:07 router kernel: module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1 Mar 13 21:47:07 router kernel: kbd0 at kbdmux0 Mar 13 21:47:07 router kernel: WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0. Mar 13 21:47:07 router kernel: netgate0: <Netgate APU>
System log before reboot has a lot of:
Mar 13 20:55:54 router kernel: arpresolve: can't allocate llinfo for 192.0.0.1 on igb1 Mar 13 20:56:04 router kernel: arpresolve: can't allocate llinfo for 192.0.0.1 on igb1
192.0.0.1 is the IPv4 IP from T-Mobile cellular I use for backup. For a deeper dive, see https://wirelessjoint.com/viewtopic.php?t=4191
Then, in resolver log:
Mar 13 20:25:28 router unbound[15698]: [15698:2] notice: ssl handshake failed 10.111.11.118 port 54946 Mar 13 20:26:50 router unbound[15698]: [15698:3] error: ssl handshake failed crypto error:0A000416:SSL routines::sslv3 alert certificate unknown Mar 13 20:26:50 router unbound[15698]: [15698:3] notice: ssl handshake failed 10.111.11.115 port 61484 Mar 13 20:52:15 router unbound[15698]: [15698:0] info: service stopped (unbound 1.18.0). Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0 Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 1: 4 queries, 4 answers from cache, 0 recursions, 3 prefetch, 0 rejected by ip ratelimiting Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0 Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0 Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 3: 3 queries, 2 answers from cache, 1 recursions, 1 prefetch, 0 rejected by ip ratelimiting Mar 13 20:52:19 router unbound[15698]: [15698:0] info: server stats for thread 3: requestlist max 1 avg 0.5 exceeded 0 jostled 0 Mar 13 20:52:19 router unbound[15698]: [15698:0] info: average recursion processing time 0.042078 sec Mar 13 20:52:19 router unbound[15698]: [15698:0] info: histogram of recursion processing times Mar 13 20:52:19 router unbound[15698]: [15698:0] info: [25%]=0 median[50%]=0 [75%]=0 Mar 13 20:52:19 router unbound[15698]: [15698:0] info: lower(secs) upper(secs) recursions Mar 13 20:52:19 router unbound[15698]: [15698:0] info: 0.032768 0.065536 1 Mar 13 20:53:04 router unbound[61026]: [61026:0] notice: init module 0: iterator Mar 13 20:53:04 router unbound[61026]: [61026:0] info: start of service (unbound 1.18.0). Mar 13 20:53:32 router unbound[61026]: [61026:0] info: service stopped (unbound 1.18.0). Mar 13 20:53:36 router unbound[61026]: [61026:0] info: server stats for thread 0: 7 queries, 1 answers from cache, 6 recursions, 0 prefetch, 0 rejected by ip ratelimiting Mar 13 20:53:36 router unbound[61026]: [61026:0] info: server stats for thread 0: requestlist max 1 avg 0.333333 exceeded 0 jostled 0 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: average recursion processing time 0.153774 sec Mar 13 20:53:36 router unbound[61026]: [61026:0] info: histogram of recursion processing times Mar 13 20:53:36 router unbound[61026]: [61026:0] info: [25%]=0.028672 median[50%]=0.174763 [75%]=0.240299 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: lower(secs) upper(secs) recursions Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.016384 0.032768 2 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.131072 0.262144 3 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.262144 0.524288 1 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: server stats for thread 1: 35 queries, 12 answers from cache, 23 recursions, 0 prefetch, 0 rejected by ip ratelimiting Mar 13 20:53:36 router unbound[61026]: [61026:0] info: server stats for thread 1: requestlist max 5 avg 1.13043 exceeded 0 jostled 0 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: average recursion processing time 0.232330 sec Mar 13 20:53:36 router unbound[61026]: [61026:0] info: histogram of recursion processing times Mar 13 20:53:36 router unbound[61026]: [61026:0] info: [25%]=0.0251611 median[50%]=0.114688 [75%]=0.199885 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: lower(secs) upper(secs) recursions Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.008192 0.016384 2 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.016384 0.032768 7 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.032768 0.065536 1 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.065536 0.131072 2 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 0.131072 0.262144 10 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: 2.000000 4.000000 1 Mar 13 20:53:36 router unbound[61026]: [61026:0] info: server stats for thread 2: 16 queries, 8 answers from cache, 8 recursions, 0 prefetch, 0 rejected by ip ratelimiting
-
Did you decide what 192.0.0.1 is?
One interesting thing is that it's detected as an APU1 but that's an APU2. What does the output of
kenv | grep smbios
show? -
@stephenw10
RE 192.0..0.1
https://wirelessjoint.com/viewtopic.php?t=4191My gut here is that the gateway pinger under pfsense is wreaking havoc if/when your IG is going between 192.168.225.x and 192.0.0.x. due to its default IPPT configuration. There are changes you can make to improve the situation with IPPT or simply disable it if you don't mind the additional NAT layer. All of this explained in detail if you read on :)
In addressing some of your issues it will first be good to shed some light on why you are seeing a 192.0.0.x address on your WAN interface even though the default UI access IP is 192.168.225.1. The InvisaGig provides IP Passthrough (IPPT) mode to the device connected to its Ethernet port by default and it uses ARP to find the connected device's MAC address which is to be assigned the IPPT address. This means that when the modem successfully connects to the carrier it bridges the IPv4 assigned by the carrier to the Ethernet interface of the InvisaGig.
If your cellular carrier network carrier uses IPv6 for the connection and you do not have a plan which specifically provides a statically assigned IPv4 address, the modem will connect to the carrier over IPv6 but perform IPv4 translation. This translation is referred to as dual stack lite, customer-side translation (CLAT) and will utilize an IPv4 address in the range defined by RFC 6333 which is typically 192.0.0.1 at the modem gateway and 192.0.0.2 assigned to the IG-connected IPPT device. In the USA, this scenario is most typical of T-Mobile consumer cellular plans and/or business plans without the static IP add-on so I am guessing this is your carrier. This is especially the case when connecting over 5G Standalone (5G SA) as that is IPv6 only. Under 5G Non-standalone (5G NSA) and LTE it is possible you may still receive a native, legacy IPv4 address from the carrier grade NAT (CGNAT) DHCP pool instead of a 192.0.0.x address.
Most devices and/or routers connected to the InvisaGig with the default IPPT functionality enabled will not have an issue with 192.0.0.x addressing on their WAN but from time to time we do see some enterprise equipment which has an issue with this address space and/or issues with IPPT bridge creation / MAC autodetection of its WAN port by the IG when reconnecting to the carrier. Depending on the root cause in the case of your PFSense router/firewall there may be a couple of settings which need changing. The first I would suggest would be to set the IPPT MAC address of the PFSense WAN port manually. -
@stephenw10
RE kenv | grep smbiossmbios.bios.reldate="02/28/2017"
smbios.bios.revision="4.0"
smbios.bios.vendor="coreboot"
smbios.bios.version="4.0.7"
smbios.chassis.maker="PC Engines"
smbios.chassis.type="Desktop"
smbios.planar.location=""
smbios.planar.maker="PC Engines"
smbios.planar.product="APU2"
smbios.planar.serial="1131876"
smbios.planar.tag=""
smbios.planar.version="1.0"
smbios.system.maker="PC Engines"
smbios.system.product="APU2"
smbios.system.serial="1131876"
smbios.system.sku="4 GB"
smbios.system.version="1.0"
smbios.version="2.7" -
Is it overheating? I had a pcie mini card reboot randomly until I added heatsinks. My logs however just showed <Boot> nothing else before it. Weird one
-
I've wondered about that, but I don't see any evidince of it. The box get good air flow, its in a cool basement. I've even placed old heat sinks on the case (not attached) to help a little. Not sure that the screenshot below is going to post, but it shows fairly steady temps of about 55°C +/- 1° (the dip is when I power cycled)
-
Ah yes DS-Lite. I see that so infrequently I always forget about it. So 192.0.0.2 is a legitimate IP address in that case.
I agree the switch from local DHCP to bridged is what I always suspected is the issue. The problem happens is the modem reboots or it;s link is lost and it starts handing out IPs in it's own subnet. When the link is re-established pfSense will not necessarily try to pull a new lease so fails until it renews.
This is quite common in cable modems and the usu workaround is to deny leases from the cable modem IP so it just keeps trying until the upstream link comes back up.
If you are seeing that it will be logged. The dhclient logs in the dhcp log will show the local modem address that's serving leases.
-
Mar 13 19:48:49 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:10 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:50 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:51:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:52:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:52:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:24 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:38 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:56 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:11 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:32 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:57:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:57:53 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:58:34 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:58:56 router dhcpd[74227]: DHCPREQUEST for 10.11.11.64 from 00:15:88:6b:bb:4d via igb2.11 Mar 13 19:58:56 router dhcpd[74227]: DHCPACK on 10.11.11.64 to 00:15:88:6b:bb:4d via igb2.11 Mar 13 19:59:28 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:00:36 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:01:32 router dhcpd[74227]: DHCPREQUEST for 10.111.11.138 from 82:58:13:8d:90:9a (Pixel-8) via igb2.111 Mar 13 20:01:32 router dhcpd[74227]: DHCPACK on 10.111.11.138 to 82:58:13:8d:90:9a (Pixel-8) via igb2.111 Mar 13 20:01:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:03:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:05:14 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:06:08 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:08:35 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:08:44 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:08:54 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:09:04 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:09:18 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:09:43 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:10:05 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:10:28 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:10:41 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:10:55 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:11:30 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:13:01 router dhcpd[74227]: DHCPREQUEST for 10.111.11.134 from 00:15:cd:5a:43:44 (0005CD5A4344) via igb2.111 Mar 13 20:13:01 router dhcpd[74227]: DHCPACK on 10.111.11.134 to 00:15:cd:5a:43:44 (0005CD5A4344) via igb2.111 Mar 13 20:13:05 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:13:53 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:14:45 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:16:01 router dhcpd[74227]: DHCPREQUEST for 10.251.11.126 from dc:h6:32:9a:88:95 (svrRPI364b) via igb2.251 Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 class decls to leases file. Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 deleted host decls to leases file. Mar 13 20:16:01 router dhcpd[74227]: Wrote 0 new dynamic host decls to leases file. Mar 13 20:16:01 router dhcpd[74227]: Wrote 129 leases to leases file. Mar 13 20:16:01 router dhcpd[74227]: DHCPACK on 10.251.11.126 to dc:h6:32:9a:88:95 (svrRPI364b) via igb2.251 Mar 13 20:16:14 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:19:42 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:21:45 router dhcpd[74227]: DHCPREQUEST for 10.111.11.130 from e6:34:b1:e6:73:8a (Pixel-7a) via igb2.111 Mar 13 20:21:45 router dhcpd[74227]: DHCPACK on 10.111.11.130 to e6:34:b1:e6:73:8a (Pixel-7a) via igb2.111 Mar 13 20:21:48 router dhclient[84101]: DHCPREQUEST on igb1 to 255.255.255.255 port 67 Mar 13 20:22:20 router dhclient[6155]: EXPIRE Mar 13 20:22:20 router dhclient[7266]: Deleting old routes Mar 13 20:22:20 router dhclient[8836]: PREINIT
Mar 13 20:54:36 router dhcpd[50201]: Server starting service. Mar 13 20:54:36 router dhclient[51958]: PREINIT Mar 13 20:54:36 router dhclient[53206]: EXPIRE Mar 13 20:54:36 router dhclient[54111]: Deleting old routes Mar 13 20:54:36 router dhclient[55441]: PREINIT Mar 13 20:54:37 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 20:54:39 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 3 Mar 13 20:54:42 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5 Mar 13 20:54:47 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11 Mar 13 20:54:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12 Mar 13 20:55:10 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15 Mar 13 20:55:25 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12 Mar 13 20:55:37 router dhclient[50960]: No DHCPOFFERS received. Mar 13 20:55:37 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 20:55:37 router dhclient[92864]: FAIL Mar 13 20:55:51 router dhcpd[82618]: Internet Systems Consortium DHCP Server 4.4.3-P1 Mar 13 20:55:51 router dhcpd[82618]: Copyright 2004-2022 Internet Systems Consortium. Mar 13 20:55:51 router dhcpd[82618]: All rights reserved. Mar 13 20:55:51 router dhcpd[82618]: For info, please visit https://www.isc.org/software/dhcp/ Mar 13 20:55:51 router dhcpd[82618]: Config file: /etc/dhcpd.conf Mar 13 20:55:51 router dhcpd[82618]: Database file: /var/db/dhcpd.leases Mar 13 20:55:51 router dhcpd[82618]: PID file: /var/run/dhcpd.pid Mar 13 20:55:51 router dhcpd[82618]: Internet Systems Consortium DHCP Server 4.4.3-P1 Mar 13 20:55:51 router dhcpd[82618]: Copyright 2004-2022 Internet Systems Consortium. Mar 13 20:55:51 router dhcpd[82618]: All rights reserved. Mar 13 20:55:51 router dhcpd[82618]: For info, please visit https://www.isc.org/software/dhcp/ Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 class decls to leases file. Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 deleted host decls to leases file. Mar 13 20:55:51 router dhcpd[82618]: Wrote 0 new dynamic host decls to leases file. Mar 13 20:55:51 router dhcpd[82618]: Wrote 129 leases to leases file. Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.101/00:0d:b9:a9:16:92/10.101.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2.101/00:0d:b9:a9:16:92/10.101.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.251/00:0d:b9:a9:16:92/10.251.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2.251/00:0d:b9:a9:16:92/10.251.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.111/00:0d:b9:a9:16:92/10.111.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2.111/00:0d:b9:a9:16:92/10.111.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.31/00:0d:b9:a9:16:92/10.31.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2.31/00:0d:b9:a9:16:92/10.31.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2.11/00:0d:b9:a9:16:92/10.11.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2.11/00:0d:b9:a9:16:92/10.11.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Listening on BPF/igb2/00:0d:b9:a9:16:92/10.0.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on BPF/igb2/00:0d:b9:a9:16:92/10.0.11.0/24 Mar 13 20:55:51 router dhcpd[82618]: Sending on Socket/fallback/fallback-net Mar 13 20:55:51 router dhcpd[82618]: Server starting service. Mar 13 20:55:52 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 20:55:54 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4 Mar 13 20:55:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6 Mar 13 20:56:04 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10 Mar 13 20:56:14 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 17 Mar 13 20:56:31 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 20 Mar 13 20:56:51 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 20:56:53 router dhclient[50960]: No DHCPOFFERS received. Mar 13 20:56:53 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 20:56:53 router dhclient[34633]: FAIL
-
Hmm, nothing shown there at all. No responses from any DHCP server. That was during some outage I assume?
-
@stephenw10
Right, that was about the time the network failure occured yesterday. I don't know exactly when it started failing, but I was troubleshooting prior to having to power cycle at 21:47 (see log in yesterday's post).Mar 13 19:44:30 router dhcpd[74227]: DHCPDISCOVER from 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31 Mar 13 19:44:30 router dhcpd[74227]: DHCPOFFER on 10.31.11.235 to 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31 Mar 13 19:44:31 router dhcpd[74227]: reuse_lease: lease age 1510 (secs) under 25% threshold, reply with unaltered, existing lease for 10.31.11.235 Mar 13 19:44:31 router dhcpd[74227]: DHCPREQUEST for 10.31.11.235 (10.31.11.1) from 30:e9:50:8e:e2:91 (Sweeper_p100) via igb2.31 Mar 13 19:48:49 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:48:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:10 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:49:50 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:50:59 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:51:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:52:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:52:54 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:04 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:24 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:31 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:53:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:29 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:39 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:54:51 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:13 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:38 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:45 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:55:56 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:11 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:22 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:32 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:56:47 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:57:17 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:57:53 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:58:34 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 19:58:56 router dhcpd[74227]: DHCPREQUEST for 10.11.11.64 from 00:15:88:6b:bb:4d via igb2.11 Mar 13 19:58:56 router dhcpd[74227]: DHCPACK on 10.11.11.64 to 00:15:88:6b:bb:4d via igb2.11 Mar 13 19:59:28 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:00:36 router dhclient[84101]: DHCPREQUEST on igb1 to 192.0.0.1 port 67 Mar 13 20:01:32 router dhcpd[74227]: DHCPREQUEST for 10.111.11.138 from 82:58:13:8d:90:9a (Pixel-8) via igb2.111 Mar 13 20:01:32 router dhcpd[74227]: DHCPACK on 10.111.11.138 to 82:58:13:8d:90:9a (Pixel-8) via igb2.111
Then is a lot of the following DHCP logs just prior to power cycle:
Mar 13 21:29:49 router dhclient[9249]: FAIL Mar 13 21:30:04 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1 Mar 13 21:30:05 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:30:07 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5 Mar 13 21:30:12 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6 Mar 13 21:30:18 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15 Mar 13 21:30:33 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 18 Mar 13 21:30:51 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12 Mar 13 21:31:03 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:31:05 router dhclient[50960]: No DHCPOFFERS received. Mar 13 21:31:05 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 21:31:05 router dhclient[53331]: FAIL Mar 13 21:31:20 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:31:22 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4 Mar 13 21:31:26 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4 Mar 13 21:31:30 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10 Mar 13 21:31:40 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13 Mar 13 21:31:53 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 16 Mar 13 21:32:06 router dhcpd[82618]: uid lease 10.111.11.126 for client 48:a9:44:91:ec:4f is duplicate on 10.111.11.0/24 Mar 13 21:32:06 router dhcpd[82618]: DHCPREQUEST for 10.111.11.51 from 48:e9:44:91:ec:4f via igb2.111 Mar 13 21:32:06 router dhcpd[82618]: DHCPACK on 10.111.11.51 to 48:a9:44:91:ec:4f via igb2.111 Mar 13 21:32:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12 Mar 13 21:32:13 router dhcpd[82618]: DHCPREQUEST for 10.0.11.121 from 78:a9:58:dd:69:cf (MstrCloset) via igb2 Mar 13 21:32:13 router dhcpd[82618]: DHCPACK on 10.0.11.121 to 78:a9:58:dd:69:cf (MstrCloset) via igb2 Mar 13 21:32:18 router dhcpd[82618]: DHCPREQUEST for 10.11.11.62 from dc:a6:32:9a:88:8c via igb2.11 Mar 13 21:32:18 router dhcpd[82618]: DHCPACK on 10.11.11.62 to dc:a6:32:9a:88:8c via igb2.11 Mar 13 21:32:21 router dhclient[50960]: No DHCPOFFERS received. Mar 13 21:32:21 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 21:32:21 router dhclient[22]: FAIL Mar 13 21:32:36 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1 Mar 13 21:32:37 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:32:39 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:32:41 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5 Mar 13 21:32:44 router dhcpd[82618]: DHCPREQUEST for 10.11.11.91 from 00:e9:9d:db:a6:54 via igb2.11 Mar 13 21:32:44 router dhcpd[82618]: DHCPACK on 10.11.11.91 to 00:e9:9d:db:a6:54 via igb2.11 Mar 13 21:32:46 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 9 Mar 13 21:32:55 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 16 Mar 13 21:33:11 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 7 Mar 13 21:33:18 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11 Mar 13 21:33:29 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 8 Mar 13 21:33:37 router dhclient[50960]: No DHCPOFFERS received. Mar 13 21:33:37 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 21:33:37 router dhclient[79182]: FAIL Mar 13 21:33:52 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:33:54 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 4 Mar 13 21:33:58 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11 Mar 13 21:34:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 11 Mar 13 21:34:20 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 12 Mar 13 21:34:32 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 15 Mar 13 21:34:47 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 6 Mar 13 21:34:53 router dhclient[50960]: No DHCPOFFERS received. Mar 13 21:34:53 router dhclient[50960]: No working leases in persistent database - sleeping. Mar 13 21:34:53 router dhclient[36776]: FAIL Mar 13 21:35:08 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 1 Mar 13 21:35:09 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2 Mar 13 21:35:11 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 5 Mar 13 21:35:16 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 10 Mar 13 21:35:18 router dhcpd[82618]: DHCPREQUEST for 10.0.11.33 from 78:a9:58:46:f9:44 via igb2 Mar 13 21:35:18 router dhcpd[82618]: DHCPACK on 10.0.11.33 to 78:a9:58:46:f9:44 via igb2 Mar 13 21:35:26 router dhclient[50960]: DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13 Mar 13 21:47:13 router dhclient[30088]: PREINIT
I just noticed this in above, which is our printer. However, there is no duplicate in the currect DHCP leases.
Mar 13 21:32:06 router dhcpd[82618]: uid lease 10.111.11.126 for client 48:a9:44:91:ec:4f is duplicate on 10.111.11.0/24 Mar 13 21:32:06 router dhcpd[82618]: DHCPREQUEST for 10.111.11.51 from 48:e9:44:91:ec:4f via igb2.111 Mar 13 21:32:06 router dhcpd[82618]: DHCPACK on 10.111.11.51 to 48:a9:44:91:ec:4f via igb2.111
-
Hmm, so it appears that when this goes down there are no responding DHCP servers on the WAN. So either the modem stops both bridging the ISP DS-Lite connection or handing out it's own leases. Or the NIC in pfSense is not actually sending the DHCP leases (or not seeing the responses).
Are you able to try to capture traffic on the WAN side?
Since you're already behind double NAT you might just try disabling the IP pass through in the modem. That would avoid the IP/subnet change. It might also prevent the modem failing if that's what is happening.
-
@stephenw10
It would take some time for me to set-up, but I could probably set-up wireshark on a RasPi & a small switch to log packets (I think thats what your suggesting.)There is a way to disable IP pass though in the modem, so I'll try that next.
-
Yep an ideal test there would be capturing packets from a switch mirror port but that's clearly quite involved to get in place.
That would prove it's not an issue with the NIC though.
-
@stephenw10 So I'm back this
I had all but decided its time to buy a new router, but had to go out of town so I threw together the script descibed below executed by cron ever 12 minutes. That was about 3 weeks ago. Router hasn't had any issue until today.The only reason it an issue was that I was testing the back-up connection by killing power the primary wan modem. Failover was fine, but when I brought the primary WAN back online the LAN comms would die. Looking at the log generated by my script, it appears the script has cycled the wan several times over the last few weeks which makes me think the script mostly works but I'm not sure why or what problem its resolving by cycling the wan.
gwstat=$(pfSsh.php playback gatewaystatus) WAN_STATE=$(echo "$gwstat" | awk '/'$GW_ID'/ { print $7}') if [ $WAN_STATE = "online" ]; then exit; fi echo "WAN Cycling on $varDate" >> $log #turn off modem using Hue Appliance Plug (doubtful this works, because lan comms are down so the Hue Hub is prob unreachable. Just now modifed script to log ouput) /usr/local/bin/curl -X PUT -H "Content-Type: application/json" -d '{"on": false}' $url >> $log ifconfig igb0 down # Sleep, then bring up modem sleep 30 /usr/local/bin/curl -X PUT -H "Content-Type: application/json" -d '{"on": true}' $url >> $log ifconfig igb0 up
-
Hmm, well if you kill the WAN connection deliberately that script is going to continually try to cycle the WAN modem. Though I'm not sure why that would prevent it coming back up or otherwise kill the LAN side connectivity.
-
One thing that was neglected to mention here, is Memtest. I’d suggest running memtest to see if there are any memory errors.
Second. Test another Power Supply, recently I had a client that had the same problem due to a faulty 12v rail.
Regards
-
Network just went down again, kind of, and through no action of mine. This time, I was able to load internet webpages through my fail-over connection (so that was working). I could ping the router over the LAN. I really don't think the primary WAN was down, as once I power cycled it was online. I also confirmed that the power plug was on (so everything was getting power and should be online.)
However, the router GUI was reporting a 504 error (white page with black text served by ngnix.) I was able SSH into the router, so I tried to restart PHP-FAM. This allowed me to load the GUI login, but after entering credentials the browser waited, waited, waited and then back to 504 error. Ended up power cycling.
Per @VioletDragon suggestion, I just replaced the power supply. Earlier in the troubleshooting process, I think I already tried another new power supply (maybe 3rd's the charm) and I had already run memtest overnight, no errors reported.
-
@Ximulate Just out of interest. Can you share System -> Advanced -> Network Interfaces.
Are you routing Layer 2 or 3 ? Is Jumbo Frames been left at default 1500 ?
Have you tested a fresh install ?
Regards.