Back to odd problem -- lose WAN at random points with a week or more between events

[Wylbur]

@Wylbur WAN failure has happened twice now. This log is from the second Failure. [I'm going throught packages to see if I missed some setting... Only thing I can think of since WAN is on Intel ethernet chipset...)

Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 23 08:44:58 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Feb 23 08:44:58 	kernel 		done.
Feb 23 08:44:58 	kernel 		pflog0: promiscuous mode enabled
Feb 23 08:44:58 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 23 08:44:58 	kernel 		.done.
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: sync unbound done.
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	check_reload_status 	473 	Linkup starting em0
Feb 23 08:44:59 	kernel 		em0: link state changed to UP
Feb 23 08:45:00 	php-fpm 	410 	/rc.linkup: Ignoring link event during boot sequence.
Feb 23 08:45:06 	kernel 		done.
Feb 23 08:45:06 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Feb 23 08:45:06 	kernel 		done.
Feb 23 08:45:07 	kernel 		done.
Feb 23 08:45:08 	kernel 		done.
Feb 23 08:45:08 	check_reload_status 	473 	Updating all dyndns
Feb 23 08:45:08 	kernel 		....
Feb 23 08:45:08 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Feb 23 08:45:08 	kernel 		.done.
Feb 23 08:45:12 	php-cgi 	508 	rc.bootup: Creating rrd update script
Feb 23 08:45:12 	syslogd 		exiting on signal 15
Feb 23 08:45:13 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 23 08:45:13 	kernel 		done.
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: Restarting/Starting all packages.
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: Starting service avahi
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: [zeek] Starting service...
Feb 23 08:45:13 	SnortStartup 	73288 	Snort START for WAN(igb1)...
Feb 23 08:45:13 	kernel 		igb1: promiscuous mode enabled
Feb 23 08:45:13 	avahi-daemon 	70713 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 23 08:45:13 	avahi-daemon 	70713 	Successfully dropped root privileges.
Feb 23 08:45:13 	avahi-daemon 	70713 	avahi-daemon 0.8 starting up.
Feb 23 08:45:13 	avahi-daemon 	70713 	No service file found in /usr/local/etc/avahi/services.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface lo0.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface em0.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface igb1.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Network interface enumeration completed.
Feb 23 08:45:13 	avahi-daemon 	70713 	Server startup complete. Host name is pfSense.local. Local service cookie is 2406046792.
Feb 23 08:45:16 	php-fpm 	410 	/rc.start_packages: [zeek] Adding cronjobs ...
Feb 23 08:45:16 	root 	96059 	Bootup complete
Feb 23 08:45:17 	login 	96637 	login on ttyv0 as root
Feb 23 08:45:17 	sshguard 	98598 	Now monitoring attacks.
Feb 23 08:45:34 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 23 10:17:00 	sshguard 	98598 	Exiting on signal.
Feb 23 10:17:00 	sshguard 	81780 	Now monitoring attacks.
Feb 23 18:45:00 	sshguard 	81780 	Exiting on signal.
Feb 23 18:45:00 	sshguard 	1738 	Now monitoring attacks.
Feb 23 20:38:00 	sshguard 	1738 	Exiting on signal.
Feb 23 20:38:00 	sshguard 	12967 	Now monitoring attacks.
Feb 24 01:46:00 	sshguard 	12967 	Exiting on signal.
Feb 24 01:46:00 	sshguard 	56276 	Now monitoring attacks.
Feb 24 08:35:00 	sshguard 	56276 	Exiting on signal.
Feb 24 08:35:00 	sshguard 	199 	Now monitoring attacks.
Feb 24 12:22:00 	sshguard 	199 	Exiting on signal.
Feb 24 12:22:00 	sshguard 	56952 	Now monitoring attacks.
Feb 24 21:00:00 	sshguard 	56952 	Exiting on signal.
Feb 24 21:00:00 	sshguard 	61484 	Now monitoring attacks.
Feb 24 21:44:59 	rc.gateway_alarm 	76930 	>>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.974ms RTTsd:.992ms Loss:21%)
Feb 24 21:44:59 	check_reload_status 	473 	updating dyndns WAN_DHCP
Feb 24 21:44:59 	check_reload_status 	473 	Restarting IPsec tunnels
Feb 24 21:44:59 	check_reload_status 	473 	Restarting OpenVPN tunnels/interfaces
Feb 24 21:44:59 	check_reload_status 	473 	Reloading filter
Feb 24 21:45:00 	sshguard 	61484 	Exiting on signal.
Feb 24 21:45:00 	sshguard 	80832 	Now monitoring attacks.
Feb 24 21:45:00 	php-fpm 	409 	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 24 21:45:00 	php-fpm 	409 	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 24 21:57:22 	php-fpm 	409 	/index.php: Session timed out for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:24 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:55 	php-fpm 	410 	/diag_reboot.php: Stopping all packages.
Feb 24 21:57:55 	SnortStartup 	72115 	Snort STOP for WAN(igb1)...
Feb 24 21:57:56 	snort 	74401 	*** Caught Term-Signal
Feb 24 21:57:59 	kernel 		igb1: promiscuous mode disabled
Feb 24 21:58:00 	php-cgi 	37048 	servicewatchdog_cron.php: Service Watchdog detected service zeek stopped. Restarting zeek (Zeek Network Security Monitor service)
Feb 24 21:58:00 	php-cgi 	37048 	servicewatchdog_cron.php: Service Watchdog detected service snort stopped. Restarting snort (Snort IDS/IPS Daemon)
Feb 24 21:58:00 	SnortStartup 	41935 	Snort START for WAN(igb1)...
Feb 24 21:58:00 	kernel 		igb1: promiscuous mode enabled
Feb 24 21:58:02 	reboot 	36197 	rebooted by root
Feb 24 21:58:02 	syslogd 		exiting on signal 15
Feb 24 21:58:55 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 24 21:58:55 	kernel 		pflog0: promiscuous mode disabled
Feb 24 21:58:55 	kernel 		igb1: promiscuous mode disabled
Feb 24 21:58:55 	kernel 		Waiting (max 60 seconds) for system process `vnlru' to stop... done
Feb 24 21:58:55 	kernel 		Waiting (max 60 seconds) for system process `syncer' to stop...
Feb 24 21:58:55 	kernel 		Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
Feb 24 21:58:55 	kernel 		All buffers synced.
Feb 24 21:58:55 	kernel 		Uptime: 1d13h13m42s
Feb 24 21:58:55 	kernel 		---<<BOOT>>---
Feb 24 21:58:55 	kernel 		Copyright (c) 1992-2023 The FreeBSD Project.
Feb 24 21:58:55 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Feb 24 21:58:55 	kernel 		The Regents of the University of California. All rights reserved.
Feb 24 21:58:55 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Feb 24 21:58:55 	kernel 		FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023
Feb 24 21:58:55 	kernel 		root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64
Feb 24 21:58:55 	kernel 		FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Feb 24 21:58:55 	kernel 		VT(efifb): resolution 1024x768
Feb 24 21:58:55 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3200.00-MHz K8-class CPU)
Feb 24 21:58:55 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Feb 24 21:58:55 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Feb 24 21:58:55 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Feb 24 21:58:55 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Feb 24 21:58:55 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Feb 24 21:58:55 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Feb 24 21:58:55 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Feb 24 21:58:55 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Feb 24 21:58:55 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Feb 24 21:58:55 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Feb 24 21:58:55 	kernel 		TSC: P-state invariant, performance statistics
Feb 24 21:58:55 	kernel 		real memory = 17179869184 (16384 MB)
Feb 24 21:58:55 	kernel 		avail memory = 16475234304 (15712 MB)
Feb 24 21:58:55 	kernel 		Event timer "LAPIC" quality 600
Feb 24 21:58:55 	kernel 		ACPI APIC Table: <HPQOEM 805D >
Feb 24 21:58:55 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Feb 24 21:58:55 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
Feb 24 21:58:55 	kernel 		random: registering fast source Intel Secure Key RNG
Feb 24 21:58:55 	kernel 		random: fast provider: "Intel Secure Key RNG"
Feb 24 21:58:55 	kernel 		random: unblocking device.
Feb 24 21:58:55 	kernel 		ioapic0 <Version 2.0> irqs 0-119
Feb 24 21:58:55 	kernel 		Launching APs: 2 1 3
Feb 24 21:58:55 	kernel 		TCP_ratelimit: Is now initialized
Feb 24 21:58:55 	kernel 		wlan: mac acl policy registered
Feb 24 21:58:55 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1
Feb 24 21:58:55 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
Feb 24 21:58:55 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1
Feb 24 21:58:55 	kernel 		random: entropy device external interface
Feb 24 21:58:55 	kernel 		kbd1 at kbdmux0
Feb 24 21:58:55 	kernel 		WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
Feb 24 21:58:55 	kernel 		netgate0: <unknown hardware>
Feb 24 21:58:55 	kernel 		efirtc0: <EFI Realtime Clock>
Feb 24 21:58:55 	kernel 		efirtc0: registered as a time-of-day clock, resolution 1.000000s
Feb 24 21:58:55 	kernel 		smbios0: <System Management BIOS> at iomem 0xc97cb000-0xc97cb01e
Feb 24 21:58:55 	kernel 		smbios0: Version: 2.7, BCD Revision: 2.7
Feb 24 21:58:55 	kernel 		acpi0: <HPQOEM SLIC-BPC>
Feb 24 21:58:55 	kernel 		acpi0: Power Button (fixed)
Feb 24 21:58:55 	kernel 		cpu0: <ACPI CPU> on acpi0
Feb 24 21:58:55 	kernel 		hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Feb 24 21:58:55 	kernel 		Timecounter "HPET" frequency 24000000 Hz quality 950
Feb 24 21:58:55 	kernel 		Event timer "HPET" frequency 24000000 Hz quality 550
Feb 24 21:58:55 	kernel 		Event timer "HPET1" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET2" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET3" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET4" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Feb 24 21:58:55 	kernel 		atrtc0: Warning: Couldn't map I/O.
Feb 24 21:58:55 	kernel 		atrtc0: registered as a time-of-day clock, resolution 1.000000s
Feb 24 21:58:55 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
Feb 24 21:58:55 	kernel 		attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Feb 24 21:58:55 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
Feb 24 21:58:55 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
Feb 24 21:58:55 	kernel 		Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Feb 24 21:58:55 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
Feb 24 21:58:55 	kernel 		acpi_ec0: <Embedded Controller: GPE 0x6e> port 0x62,0x66 on acpi0
Feb 24 21:58:55 	kernel 		pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Feb 24 21:58:55 	kernel 		pci0: <ACPI PCI bus> on pcib0
Feb 24 21:58:55 	kernel 		vgapci0: <VGA-compatible display> port 0x4000-0x403f mem 0xe0000000-0xe0ffffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
Feb 24 21:58:55 	kernel 		vgapci0: Boot video device
Feb 24 21:58:55 	kernel 		xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xe1320000-0xe132ffff at device 20.0 on pci0
Feb 24 21:58:55 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
Feb 24 21:58:55 	kernel 		usbus0 on xhci0
Feb 24 21:58:55 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
Feb 24 21:58:55 	kernel 		pci0: <simple comms> at device 22.0 (no driver attached)
Feb 24 21:58:55 	kernel 		uart2: <100 Series/C230 Series Chipset Family KT Redirection> port 0x4080-0x4087 mem 0xe134a000-0xe134afff at device 22.3 on pci0
Feb 24 21:58:55 	kernel 		ahci0: <Intel Sunrise Point AHCI SATA controller> port 0x4088-0x408f,0x4090-0x4093,0x4040-0x405f mem 0xe1348000-0xe1349fff,0xe134f000-0xe134f0ff,0xe134d000-0xe134d7ff at device 23.0 on pci0
Feb 24 21:58:55 	kernel 		ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
Feb 24 21:58:55 	kernel 		ahcich0: <AHCI channel> at channel 0 on ahci0
Feb 24 21:58:55 	kernel 		ahcich1: <AHCI channel> at channel 1 on ahci0
Feb 24 21:58:55 	kernel 		ahcich2: <AHCI channel> at channel 2 on ahci0
Feb 24 21:58:55 	kernel 		ahcich3: <AHCI channel> at channel 3 on ahci0
Feb 24 21:58:55 	kernel 		ahcich4: <AHCI channel> at channel 4 on ahci0
Feb 24 21:58:55 	kernel 		ahcich5: <AHCI channel> at channel 5 on ahci0
Feb 24 21:58:55 	kernel 		ahciem0: <AHCI enclosure management bridge> on ahci0
Feb 24 21:58:55 	kernel 		pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
Feb 24 21:58:55 	kernel 		pci1: <ACPI PCI bus> on pcib1
Feb 24 21:58:55 	kernel 		igb0: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3000-0x301f mem 0xe1200000-0xe121ffff,0xe1000000-0xe10fffff,0xe1240000-0xe1243fff at device 0.0 on pci1
Feb 24 21:58:55 	kernel 		igb0: EEPROM V1.13-1
Feb 24 21:58:55 	kernel 		igb0: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		igb0: Using 4 RX queues 4 TX queues
Feb 24 21:58:55 	kernel 		igb0: Using MSI-X interrupts with 5 vectors
Feb 24 21:58:55 	kernel 		igb0: Ethernet address: 00:1b:21:27:12:36
Feb 24 21:58:55 	kernel 		igb0: netmap queues/slots: TX 4/1024, RX 4/1024
Feb 24 21:58:55 	kernel 		igb1: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3020-0x303f mem 0xe1220000-0xe123ffff,0xe1100000-0xe11fffff,0xe1244000-0xe1247fff at device 0.1 on pci1
Feb 24 21:58:55 	kernel 		igb1: EEPROM V1.13-1
Feb 24 21:58:55 	kernel 		igb1: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		igb1: Using 4 RX queues 4 TX queues
Feb 24 21:58:55 	kernel 		igb1: Using MSI-X interrupts with 5 vectors
Feb 24 21:58:55 	kernel 		igb1: Ethernet address: 00:1b:21:27:12:37
Feb 24 21:58:55 	kernel 		igb1: netmap queues/slots: TX 4/1024, RX 4/1024
Feb 24 21:58:55 	kernel 		isab0: <PCI-ISA bridge> at device 31.0 on pci0
Feb 24 21:58:55 	kernel 		isa0: <ISA bus> on isab0
Feb 24 21:58:55 	kernel 		pci0: <memory> at device 31.2 (no driver attached)
Feb 24 21:58:55 	kernel 		hdac0: <Intel Sunrise Point HDA Controller> mem 0xe1344000-0xe1347fff,0xe1330000-0xe133ffff at device 31.3 on pci0
Feb 24 21:58:55 	kernel 		em0: <Intel(R) I219-LM SPT-H(2)> mem 0xe1300000-0xe131ffff at device 31.6 on pci0
Feb 24 21:58:55 	kernel 		em0: EEPROM V0.8-4
Feb 24 21:58:55 	kernel 		em0: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		em0: Using an MSI interrupt
Feb 24 21:58:55 	kernel 		em0: Ethernet address: 70:5a:0f:3b:19:12
Feb 24 21:58:55 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
Feb 24 21:58:55 	kernel 		acpi_button0: <Sleep Button> on acpi0
Feb 24 21:58:55 	kernel 		acpi_button1: <Power Button> on acpi0
Feb 24 21:58:55 	kernel 		atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Feb 24 21:58:55 	kernel 		atkbd0: <AT Keyboard> irq 1 on atkbdc0
Feb 24 21:58:55 	kernel 		kbd0 at atkbd0
Feb 24 21:58:55 	kernel 		atkbd0: [GIANT-LOCKED]
Feb 24 21:58:55 	kernel 		acpi_syscontainer0: <System Container> on acpi0
Feb 24 21:58:55 	kernel 		uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Feb 24 21:58:55 	kernel 		hwpstate_intel0: <Intel Speed Shift> on cpu0
Feb 24 21:58:55 	kernel 		hwpstate_intel1: <Intel Speed Shift> on cpu1
Feb 24 21:58:55 	kernel 		hwpstate_intel2: <Intel Speed Shift> on cpu2
Feb 24 21:58:55 	kernel 		hwpstate_intel3: <Intel Speed Shift> on cpu3
Feb 24 21:58:55 	kernel 		Timecounter "TSC-low" frequency 1595996789 Hz quality 1000
Feb 24 21:58:55 	kernel 		Timecounters tick every 1.000 msec
Feb 24 21:58:55 	kernel 		ZFS filesystem version: 5
Feb 24 21:58:55 	kernel 		ZFS storage pool version: features support (5000)
Feb 24 21:58:55 	kernel 		hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Feb 24 21:58:55 	kernel 		hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Feb 24 21:58:55 	kernel 		pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Feb 24 21:58:55 	kernel 		pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Feb 24 21:58:55 	kernel 		hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Feb 24 21:58:55 	kernel 		hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Feb 24 21:58:55 	kernel 		pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1
Feb 24 21:58:55 	kernel 		Trying to mount root from zfs:pfSense/ROOT/default []...
Feb 24 21:58:55 	kernel 		ugen0.1: <Intel XHCI root HUB> at usbus0
Feb 24 21:58:55 	kernel 		uhub0 on usbus0
Feb 24 21:58:55 	kernel 		uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Feb 24 21:58:55 	kernel 		Root mount waiting for: usbus0 CAM
Feb 24 21:58:55 	kernel 		uhub0: 24 ports with 24 removable, self powered
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
Feb 24 21:58:55 	kernel 		ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Feb 24 21:58:55 	kernel 		ses0: SEMB SES Device
Feb 24 21:58:55 	kernel 		ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Feb 24 21:58:55 	kernel 		ada0: <KingFast HP2818E3> ACS-2 ATA SATA 3.x device
Feb 24 21:58:55 	kernel 		ada0: Serial Number CI0311M000805
Feb 24 21:58:55 	kernel 		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 4096bytes)
Feb 24 21:58:55 	kernel 		ada0: Command Queueing enabled
Feb 24 21:58:55 	kernel 		ada0: 244198MB (500118192 512 byte sectors)
Feb 24 21:58:55 	kernel 		ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Feb 24 21:58:55 	kernel 		ada1: <SATA SSD SBFM61.5> ACS-4 ATA SATA 3.x device
Feb 24 21:58:55 	kernel 		ada1: Serial Number 23052925600661
Feb 24 21:58:55 	kernel 		ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 24 21:58:55 	kernel 		ada1: Command Queueing enabled
Feb 24 21:58:55 	kernel 		ada1: 244198MB (500118192 512 byte sectors)
Feb 24 21:58:55 	kernel 		ses0: pass0,ada0 in 'Slot 00', SATA Slot: scbus0 target 0
Feb 24 21:58:55 	kernel 		ses0: pass1,ada1 in 'Slot 01', SATA Slot: scbus1 target 0
Feb 24 21:58:55 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
Feb 24 21:58:55 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Feb 24 21:58:55 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Feb 24 21:58:55 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Feb 24 21:58:55 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Feb 24 21:58:55 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Feb 24 21:58:55 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Feb 24 21:58:55 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Feb 24 21:58:55 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Feb 24 21:58:55 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Feb 24 21:58:55 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Feb 24 21:58:55 	kernel 		TSC: P-state invariant, performance statistics
Feb 24 21:58:55 	kernel 		pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xe134b000-0xe134bfff at device 20.2 on pci0
Feb 24 21:58:55 	kernel 		ichsmb0: <Intel Sunrise Point-H SMBus controller> port 0xefa0-0xefbf mem 0xe134e000-0xe134e0ff at device 31.4 on pci0
Feb 24 21:58:55 	kernel 		smbus0: <System Management Bus> on ichsmb0
Feb 24 21:58:55 	kernel 		lo0: link state changed to UP
Feb 24 21:58:58 	check_reload_status 	473 	Linkup starting igb1
Feb 24 21:58:58 	kernel 		igb1: link state changed to UP
Feb 24 21:59:02 	check_reload_status 	473 	rc.newwanip starting igb1
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 24 21:59:04 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Feb 24 21:59:04 	kernel 		done.
Feb 24 21:59:04 	kernel 		pflog0: promiscuous mode enabled
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 24 21:59:05 	kernel 		.done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: sync unbound done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:06 	check_reload_status 	473 	Linkup starting em0
Feb 24 21:59:06 	kernel 		em0: link state changed to UP
Feb 24 21:59:07 	php-fpm 	410 	/rc.linkup: Ignoring link event during boot sequence.
Feb 24 21:59:11 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Feb 24 21:59:11 	kernel 		done.
Feb 24 21:59:12 	kernel 		done.
Feb 24 21:59:12 	kernel 		done.
Feb 24 21:59:12 	check_reload_status 	473 	Updating all dyndns
Feb 24 21:59:13 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Feb 24 21:59:13 	kernel 		.done.
Feb 24 21:59:17 	php-cgi 	508 	rc.bootup: Creating rrd update script
Feb 24 21:59:17 	kernel 		done.
Feb 24 21:59:17 	syslogd 		exiting on signal 15
Feb 24 21:59:17 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: Restarting/Starting all packages.
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: Starting service avahi
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: [zeek] Starting service...
Feb 24 21:59:17 	SnortStartup 	58538 	Snort START for WAN(igb1)...
Feb 24 21:59:17 	avahi-daemon 	55968 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 24 21:59:17 	kernel 		igb1: promiscuous mode enabled
Feb 24 21:59:17 	avahi-daemon 	55968 	Successfully dropped root privileges.
Feb 24 21:59:17 	avahi-daemon 	55968 	avahi-daemon 0.8 starting up.
Feb 24 21:59:17 	avahi-daemon 	55968 	No service file found in /usr/local/etc/avahi/services.
Feb 24 21:59:17 	avahi-daemon 	55968 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Feb 24 21:59:17 	avahi-daemon 	55968 	New relevant interface lo0.IPv4 for mDNS.
Feb 24 21:59:17 	avahi-daemon 	55968 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Feb 24 21:59:18 	avahi-daemon 	55968 	New relevant interface em0.IPv4 for mDNS.
Feb 24 21:59:18 	avahi-daemon 	55968 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Feb 24 21:59:18 	avahi-daemon 	55968 	New relevant interface igb1.IPv4 for mDNS.
Feb 24 21:59:18 	avahi-daemon 	55968 	Network interface enumeration completed.
Feb 24 21:59:18 	avahi-daemon 	55968 	Server startup complete. Host name is pfSense.local. Local service cookie is 490430812.
Feb 24 21:59:20 	php-fpm 	410 	/rc.start_packages: [zeek] Adding cronjobs ...
Feb 24 21:59:20 	root 	80968 	Bootup complete
Feb 24 21:59:21 	login 	91105 	login on ttyv0 as root
Feb 24 21:59:21 	sshguard 	92617 	Now monitoring attacks.
Feb 24 22:00:44 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)

[stephenw10]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

Feb 24 21:00:00 sshguard 61484 Now monitoring attacks.
Feb 24 21:44:59 rc.gateway_alarm 76930 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.974ms RTTsd:.992ms Loss:21%)
Feb 24 21:44:59 check_reload_status 473 updating dyndns WAN_DHCP
Feb 24 21:44:59 check_reload_status 473 Restarting IPsec tunnels
Feb 24 21:44:59 check_reload_status 473 Restarting OpenVPN tunnels/interfaces
Feb 24 21:44:59 check_reload_status 473 Reloading filter
Feb 24 21:45:00 sshguard 61484 Exiting on signal.
Feb 24 21:45:00 sshguard 80832 Now monitoring attacks.
Feb 24 21:45:00 php-fpm 409 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 24 21:45:00 php-fpm 409 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 24 21:57:22 php-fpm 409 /index.php: Session timed out for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:24 php-fpm 409 /index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:55 php-fpm 410 /diag_reboot.php: Stopping all packages.

Ok so it doesn't lose link on WAN. It looks like the WAN simply stops passing traffic.

I would check the ARP table when the WAN fails, see if the gateway still shows.

Then run a packet capture on the WAN interface and see what's actually happening at the time.

Steve

[Wylbur]

@stephenw10

New symptoms. To get us on the same page, this is the main pfSense server (new built system). This hardware/system was used for the last report.

"Error 5, Unretryable error" I've not seen this before...... [copied this out of the log that is below] --- could this be related to time change?

Meanwhile, I had an iPhone streaming TuneIn from Ireland (from yesterday afternoon) when the following happened and it was still streaming until I forced a power down for reboot of the pfSense server (I could not log into it, ping of the 192.168.1.1 (pfSense) failed, which explains why I couldn't login as admin to issue commands). I didn't think to use a wired keyboard & mouse -- was not quite awake. So I also couldn't get a packet trace or reboot before doing a forced power down and restart.

This is what I captured in the logs once I got the system rebooted:

Mar 9 06:38:00 	sshguard 	44643 	Exiting on signal.
Mar 9 06:38:00 	sshguard 	92365 	Now monitoring attacks.
Mar 9 17:46:00 	sshguard 	92365 	Exiting on signal.
Mar 9 17:46:00 	sshguard 	4750 	Now monitoring attacks.
Mar 9 18:45:00 	sshguard 	4750 	Exiting on signal.
Mar 9 18:45:00 	sshguard 	15612 	Now monitoring attacks.
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 38 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 f8 17 e9 40 08 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 28 e0 19 ab 40 0a 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2a 28 40 00 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2c cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2e cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 30 40 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 70 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 09:08:42 	syslogd 		kernel boot file is /boot/kernel/kernel
Mar 10 09:08:42 	kernel 		---<<BOOT>>---
Mar 10 09:08:42 	kernel 		Copyright (c) 1992-2023 The FreeBSD Project.
Mar 10 09:08:42 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Mar 10 09:08:42 	kernel 		The Regents of the University of California. All rights reserved.
Mar 10 09:08:42 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Mar 10 09:08:42 	kernel 		FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023
Mar 10 09:08:42 	kernel 		root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64
Mar 10 09:08:42 	kernel 		FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Mar 10 09:08:42 	kernel 		VT(efifb): resolution 1024x768
Mar 10 09:08:42 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3200.00-MHz K8-class CPU)
Mar 10 09:08:42 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Mar 10 09:08:42 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Mar 10 09:08:42 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Mar 10 09:08:42 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Mar 10 09:08:42 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Mar 10 09:08:42 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Mar 10 09:08:42 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Mar 10 09:08:42 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Mar 10 09:08:42 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Mar 10 09:08:42 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Mar 10 09:08:42 	kernel 		TSC: P-state invariant, performance statistics
Mar 10 09:08:42 	kernel 		real memory = 17179869184 (16384 MB)
Mar 10 09:08:42 	kernel 		avail memory = 16475234304 (15712 MB)
Mar 10 09:08:42 	kernel 		Event timer "LAPIC" quality 600
Mar 10 09:08:42 	kernel 		ACPI APIC Table: <HPQOEM 805D >
Mar 10 09:08:42 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Mar 10 09:08:42 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
Mar 10 09:08:42 	kernel 		random: registering fast source Intel Secure Key RNG
Mar 10 09:08:42 	kernel 		random: fast provider: "Intel Secure Key RNG"
Mar 10 09:08:42 	kernel 		random: unblocking device.
Mar 10 09:08:42 	kernel 		ioapic0 <Version 2.0> irqs 0-119
Mar 10 09:08:42 	kernel 		Launching APs: 3 2 1
Mar 10 09:08:42 	kernel 		TCP_ratelimit: Is now initialized
Mar 10 09:08:42 	kernel 		wlan: mac acl policy registered
Mar 10 09:08:42 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1
Mar 10 09:08:42 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
Mar 10 09:08:42 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1
Mar 10 09:08:42 	kernel 		random: entropy device external interface
Mar 10 09:08:42 	kernel 		kbd1 at kbdmux0
Mar 10 09:08:42 	kernel 		WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
Mar 10 09:08:42 	kernel 		netgate0: <unknown hardware>
Mar 10 09:08:42 	kernel 		efirtc0: <EFI Realtime Clock>
Mar 10 09:08:42 	kernel 		efirtc0: registered as a time-of-day clock, resolution 1.000000s
Mar 10 09:08:42 	kernel 		smbios0: <System Management BIOS> at iomem 0xc97cb000-0xc97cb01e
Mar 10 09:08:42 	kernel 		smbios0: Version: 2.7, BCD Revision: 2.7
Mar 10 09:08:42 	kernel 		acpi0: <HPQOEM SLIC-BPC>
Mar 10 09:08:42 	kernel 		acpi0: Power Button (fixed)
Mar 10 09:08:42 	kernel 		cpu0: <ACPI CPU> on acpi0
Mar 10 09:08:42 	kernel 		hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Mar 10 09:08:42 	kernel 		Timecounter "HPET" frequency 24000000 Hz quality 950
Mar 10 09:08:42 	kernel 		Event timer "HPET" frequency 24000000 Hz quality 550
Mar 10 09:08:42 	kernel 		Event timer "HPET1" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET2" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET3" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET4" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Mar 10 09:08:42 	kernel 		atrtc0: Warning: Couldn't map I/O.
Mar 10 09:08:42 	kernel 		atrtc0: registered as a time-of-day clock, resolution 1.000000s
Mar 10 09:08:42 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
Mar 10 09:08:42 	kernel 		attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Mar 10 09:08:42 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
Mar 10 09:08:42 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
Mar 10 09:08:42 	kernel 		Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Mar 10 09:08:42 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
Mar 10 09:08:42 	kernel 		acpi_ec0: <Embedded Controller: GPE 0x6e> port 0x62,0x66 on acpi0
Mar 10 09:08:42 	kernel 		pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Mar 10 09:08:42 	kernel 		pci0: <ACPI PCI bus> on pcib0
Mar 10 09:08:42 	kernel 		vgapci0: <VGA-compatible display> port 0x4000-0x403f mem 0xe0000000-0xe0ffffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
Mar 10 09:08:42 	kernel 		vgapci0: Boot video device
Mar 10 09:08:42 	kernel 		xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xe1320000-0xe132ffff at device 20.0 on pci0
Mar 10 09:08:42 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
Mar 10 09:08:42 	kernel 		usbus0 on xhci0
Mar 10 09:08:42 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
Mar 10 09:08:42 	kernel 		pci0: <simple comms> at device 22.0 (no driver attached)
Mar 10 09:08:42 	kernel 		uart2: <100 Series/C230 Series Chipset Family KT Redirection> port 0x4080-0x4087 mem 0xe134a000-0xe134afff at device 22.3 on pci0
Mar 10 09:08:42 	kernel 		ahci0: <Intel Sunrise Point AHCI SATA controller> port 0x4088-0x408f,0x4090-0x4093,0x4040-0x405f mem 0xe1348000-0xe1349fff,0xe134f000-0xe134f0ff,0xe134d000-0xe134d7ff at device 23.0 on pci0
Mar 10 09:08:42 	kernel 		ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
Mar 10 09:08:42 	kernel 		ahcich0: <AHCI channel> at channel 0 on ahci0
Mar 10 09:08:42 	kernel 		ahcich1: <AHCI channel> at channel 1 on ahci0
Mar 10 09:08:42 	kernel 		ahcich2: <AHCI channel> at channel 2 on ahci0
Mar 10 09:08:42 	kernel 		ahcich3: <AHCI channel> at channel 3 on ahci0
Mar 10 09:08:42 	kernel 		ahcich4: <AHCI channel> at channel 4 on ahci0
Mar 10 09:08:42 	kernel 		ahcich5: <AHCI channel> at channel 5 on ahci0
Mar 10 09:08:42 	kernel 		ahciem0: <AHCI enclosure management bridge> on ahci0
Mar 10 09:08:42 	kernel 		pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
Mar 10 09:08:42 	kernel 		pci1: <ACPI PCI bus> on pcib1
Mar 10 09:08:42 	kernel 		igb0: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3000-0x301f mem 0xe1200000-0xe121ffff,0xe1000000-0xe10fffff,0xe1240000-0xe1243fff at device 0.0 on pci1
Mar 10 09:08:42 	kernel 		igb0: EEPROM V1.13-1
Mar 10 09:08:42 	kernel 		igb0: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		igb0: Using 4 RX queues 4 TX queues
Mar 10 09:08:42 	kernel 		igb0: Using MSI-X interrupts with 5 vectors
Mar 10 09:08:42 	kernel 		igb0: Ethernet address: 00:1b:21:27:12:36
Mar 10 09:08:42 	kernel 		igb0: netmap queues/slots: TX 4/1024, RX 4/1024
Mar 10 09:08:42 	kernel 		igb1: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3020-0x303f mem 0xe1220000-0xe123ffff,0xe1100000-0xe11fffff,0xe1244000-0xe1247fff at device 0.1 on pci1
Mar 10 09:08:42 	kernel 		igb1: EEPROM V1.13-1
Mar 10 09:08:42 	kernel 		igb1: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		igb1: Using 4 RX queues 4 TX queues
Mar 10 09:08:42 	kernel 		igb1: Using MSI-X interrupts with 5 vectors
Mar 10 09:08:42 	kernel 		igb1: Ethernet address: 00:1b:21:27:12:37
Mar 10 09:08:42 	kernel 		igb1: netmap queues/slots: TX 4/1024, RX 4/1024
Mar 10 09:08:42 	kernel 		isab0: <PCI-ISA bridge> at device 31.0 on pci0
Mar 10 09:08:42 	kernel 		isa0: <ISA bus> on isab0
Mar 10 09:08:42 	kernel 		pci0: <memory> at device 31.2 (no driver attached)
Mar 10 09:08:42 	kernel 		hdac0: <Intel Sunrise Point HDA Controller> mem 0xe1344000-0xe1347fff,0xe1330000-0xe133ffff at device 31.3 on pci0
Mar 10 09:08:42 	kernel 		em0: <Intel(R) I219-LM SPT-H(2)> mem 0xe1300000-0xe131ffff at device 31.6 on pci0
Mar 10 09:08:42 	kernel 		em0: EEPROM V0.8-4
Mar 10 09:08:42 	kernel 		em0: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		em0: Using an MSI interrupt
Mar 10 09:08:42 	kernel 		em0: Ethernet address: 70:5a:0f:3b:19:12
Mar 10 09:08:42 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
Mar 10 09:08:42 	kernel 		acpi_button0: <Sleep Button> on acpi0
Mar 10 09:08:42 	kernel 		acpi_button1: <Power Button> on acpi0
Mar 10 09:08:42 	kernel 		atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Mar 10 09:08:42 	kernel 		atkbd0: <AT Keyboard> irq 1 on atkbdc0
Mar 10 09:08:42 	kernel 		kbd0 at atkbd0
Mar 10 09:08:42 	kernel 		atkbd0: [GIANT-LOCKED]
Mar 10 09:08:42 	kernel 		acpi_syscontainer0: <System Container> on acpi0
Mar 10 09:08:42 	kernel 		uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Mar 10 09:08:42 	kernel 		hwpstate_intel0: <Intel Speed Shift> on cpu0
Mar 10 09:08:42 	kernel 		hwpstate_intel1: <Intel Speed Shift> on cpu1
Mar 10 09:08:42 	kernel 		hwpstate_intel2: <Intel Speed Shift> on cpu2
Mar 10 09:08:42 	kernel 		hwpstate_intel3: <Intel Speed Shift> on cpu3
Mar 10 09:08:42 	kernel 		Timecounter "TSC-low" frequency 1595996890 Hz quality 1000
Mar 10 09:08:42 	kernel 		Timecounters tick every 1.000 msec
Mar 10 09:08:42 	kernel 		ZFS filesystem version: 5
Mar 10 09:08:42 	kernel 		ZFS storage pool version: features support (5000)
Mar 10 09:08:42 	kernel 		hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Mar 10 09:08:42 	kernel 		hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Mar 10 09:08:42 	kernel 		pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Mar 10 09:08:42 	kernel 		pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Mar 10 09:08:42 	kernel 		hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Mar 10 09:08:42 	kernel 		hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Mar 10 09:08:42 	kernel 		pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1
Mar 10 09:08:42 	kernel 		Trying to mount root from zfs:pfSense/ROOT/default []...
Mar 10 09:08:42 	kernel 		ugen0.1: <Intel XHCI root HUB> at usbus0
Mar 10 09:08:42 	kernel 		uhub0 on usbus0
Mar 10 09:08:42 	kernel 		uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Mar 10 09:08:42 	kernel 		Root mount waiting for: usbus0 CAM
Mar 10 09:08:42 	kernel 		uhub0: 24 ports with 24 removable, self powered
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
Mar 10 09:08:42 	kernel 		ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Mar 10 09:08:42 	kernel 		ses0: SEMB SES Device
Mar 10 09:08:42 	kernel 		ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Mar 10 09:08:42 	kernel 		ada0: <KingFast HP2818E3> ACS-2 ATA SATA 3.x device
Mar 10 09:08:42 	kernel 		ada0: Serial Number CI0311M000805
Mar 10 09:08:42 	kernel 		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 4096bytes)
Mar 10 09:08:42 	kernel 		ada0: Command Queueing enabled
Mar 10 09:08:42 	kernel 		ada0: 244198MB (500118192 512 byte sectors)
Mar 10 09:08:42 	kernel 		ses0: pass0,ada0 in 'Slot 00', SATA Slot: scbus0 target 0
Mar 10 09:08:42 	kernel 		ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Mar 10 09:08:42 	kernel 		ada1: <SATA SSD SBFM61.5> ACS-4 ATA SATA 3.x device
Mar 10 09:08:42 	kernel 		ada1: Serial Number 23052925600661
Mar 10 09:08:42 	kernel 		ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Mar 10 09:08:42 	kernel 		ada1: Command Queueing enabled
Mar 10 09:08:42 	kernel 		ada1: 244198MB (500118192 512 byte sectors)
Mar 10 09:08:42 	kernel 		ses0: pass1,ada1 in 'Slot 01', SATA Slot: scbus1 target 0
Mar 10 09:08:42 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
Mar 10 09:08:42 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Mar 10 09:08:42 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Mar 10 09:08:42 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Mar 10 09:08:42 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Mar 10 09:08:42 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Mar 10 09:08:42 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Mar 10 09:08:42 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Mar 10 09:08:42 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Mar 10 09:08:42 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Mar 10 09:08:42 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Mar 10 09:08:42 	kernel 		TSC: P-state invariant, performance statistics
Mar 10 09:08:42 	kernel 		pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xe134b000-0xe134bfff at device 20.2 on pci0
Mar 10 09:08:42 	kernel 		ichsmb0: <Intel Sunrise Point-H SMBus controller> port 0xefa0-0xefbf mem 0xe134e000-0xe134e0ff at device 31.4 on pci0
Mar 10 09:08:42 	kernel 		smbus0: <System Management Bus> on ichsmb0
Mar 10 09:08:42 	kernel 		lo0: link state changed to UP
Mar 10 09:08:45 	check_reload_status 	473 	Linkup starting igb1
Mar 10 09:08:45 	kernel 		igb1: link state changed to UP
Mar 10 09:08:51 	check_reload_status 	473 	rc.newwanip starting igb1
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Mar 10 09:08:53 	kernel 		[fib_algo] inet.0 (bsearch4#20) rebuild_fd_flm: switching algo to radix4_lockless
Mar 10 09:08:53 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Mar 10 09:08:54 	kernel 		pflog0: promiscuous mode enabled
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Mar 10 09:08:54 	kernel 		.done.
Mar 10 09:08:54 	kernel 		done.
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	php-cgi 	508 	rc.bootup: sync unbound done.
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	check_reload_status 	473 	Linkup starting em0
Mar 10 09:08:55 	kernel 		em0: link state changed to UP
Mar 10 09:08:56 	php-fpm 	409 	/rc.linkup: Ignoring link event during boot sequence.
Mar 10 09:09:02 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Mar 10 09:09:02 	kernel 		done.
Mar 10 09:09:03 	kernel 		done.
Mar 10 09:09:03 	kernel 		done.
Mar 10 09:09:03 	check_reload_status 	473 	Updating all dyndns
Mar 10 09:09:03 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Mar 10 09:09:03 	kernel 		.done.
Mar 10 09:09:08 	php-cgi 	508 	rc.bootup: Creating rrd update script
Mar 10 09:09:08 	kernel 		done.
Mar 10 09:09:08 	syslogd 		exiting on signal 15
Mar 10 09:09:08 	syslogd 		kernel boot file is /boot/kernel/kernel
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: Restarting/Starting all packages.
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: Starting service avahi
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: [zeek] Starting service...
Mar 10 09:09:09 	SnortStartup 	85723 	Snort START for WAN(igb1)...
Mar 10 09:09:09 	avahi-daemon 	82896 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Mar 10 09:09:09 	kernel 		igb1: promiscuous mode enabled
Mar 10 09:09:09 	avahi-daemon 	82896 	Successfully dropped root privileges.
Mar 10 09:09:09 	avahi-daemon 	82896 	avahi-daemon 0.8 starting up.
Mar 10 09:09:09 	avahi-daemon 	82896 	No service file found in /usr/local/etc/avahi/services.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface lo0.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface em0.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface igb1.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Network interface enumeration completed.
Mar 10 09:09:09 	avahi-daemon 	82896 	Server startup complete. Host name is pfSense.local. Local service cookie is 804603802.
Mar 10 09:09:12 	php-fpm 	409 	/rc.start_packages: [zeek] Adding cronjobs ...
Mar 10 09:09:14 	root 	57312 	Bootup complete
Mar 10 09:09:15 	login 	58466 	login on ttyv0 as root
Mar 10 09:09:15 	sshguard 	60700 	Now monitoring attacks.
Mar 10 09:09:17 	arpwatch 	87523 	listening on igb1
Mar 10 09:09:34 	php-fpm 	410 	/diag_packet_capture.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database) 

---

And the network appears to be funtioning correctly now.

[Sergei_Shablovsky]

Sorry to insert my 5c on this, but:

BUY used/new Intel-based NIC FROM WELL-KNOWN DEVELOPER (Intel, IBM preferable) on eBay/local store.

on pfSense

• RESET BIOS/UEFI to default
  UPGRADE both(!, upper and lower) IMAGE OF BIOS/UEFI from official MB’s manufacturer web;
• PULL all PSU OUT (if they are hot swap), internal CR2032 battery, wait 1min, take it all in place back;
• DISABLE ALL POWER MANAGEMENT ON pfSense’s MOTHERBOARD (especially for CPU, PCI and NICs)!
  COLD Restart.

REPLACE all NICs that installed in main pfSense.
Starting Kali Linux (or Win10/11) from USB-drive, and:

• check that ALL NICs able to receive IPs from uplink ISP;
• check line rate/error free on each NIC by iperf3 public servers;

As a result on this point You are known that NICs and cables are on good working order.

INSTALL FRESH pfSense with option “Use the previous configuration file”.

RE-ASSIGN INTERFACES to new in local-attached keyb/monitor, COM-port terminal, or WebGUI.

COLD RESTART.

And see if issue still exist.

P.S.
You spend so much time on searching on this forum, replying, googling the same issue, so MUCH FASTER would be buying the few new NICs (ok, but it used;) check hardware first.
Anyway, You not be disappointed by this NICs upgrade in a future.

[bmeeks]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 38 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 f8 17 e9 40 08 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 28 e0 19 ab 40 0a 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2a 28 40 00 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2c cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2e cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 30 40 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 70 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 09:08:42 syslogd kernel boot file is /boot/kernel/kernel
Mar 10 09:08:42 kernel ---<<BOOT>>---

These errors indicate a failing disk drive (whether it's an SSD or an old spinning surface, it is failing).

You need to be sure you have a backup of the firewall configuration on separate media (such as a USB stick), then replace the failing drive and reinstall pfSense from an install image restoring you config during the install process.

[keyser]

@Wylbur I think you need to have a look in the DHCP log and see if the issue arises when DHCLIENT (WAN DHCP client) tries to renew the DHCP lease. Some ISP’s are quite picky with other hardware on their infrastructure, and require a quite strict DHCP client configuration.
You know that your DISCOVER/OFFER/REQUEST/ACK (new DHCP lease works), but does a renewal of an existing lease?

[Wylbur]

@Sergei_Shablovsky

Thank you for your input. But, I've already done that. This is what the WAN port is running with. The LAN port is whatever the MOBO has and I never seem to have problems with that port. The weirdness is, this MOBO will not accept connections on both the Intel ports of the dual Intel port ethernet adapter that I'm using on this machine. Yet when I ran that adapter in another machine, both ports were usable so one was for WAN and the other for LAN.

But we have to ask this question: Why was I able to run for months on end when using Realtek ports with Untangle, or IPfire with this ISP? That is the thing that puzzels me.

Now, if I were a "c" (or assembly language) programmer and knew the x86 architecture as well as I do z/Architecture machines (IBM Mainframes), I could probably code a trap to capture this failure and know why it was happening. Or I could run a trace of it that we could examine once it failed. But I don't know this architecture at that level. So unfortunately, I'm more of a knowledgeable user that knows enough to just be dangerous.

Wylbur

[Wylbur]

@keyser

I see no failures that indicate a problem with renewal of lease with the ISP. What I do see are some changes where the fiber optic modem may get its IPv4 IP address changed and then the WAN is given a new IPv4 address. And then some 8.8.8.8 pings take place and some latency is noted.

Now and then I see alerts for latency with the ISP against 8.8.8.8 and the system recovers.

What should I be looking for in that would show me I have the problem you are suspecting? I've been scanning logs off and on for weeks looking for anomalies that would tell me something. Meanwhile on these latency issues, we know that the ISP has the ability to run Gigabit connections. What we have is 200/200 Mbs. And I generally have no stuttering within my Lan with this. And I have multiple devices streaming. I am constantly listening to European radio via tunein (old iphone) which tells me pretty quicly if I've just lost connections.

Wylbur

[keyser]

@Wylbur You would know from the logs if renew was failing, because the logs would fill with a lot of renew attempts (with an increasing timer). So thats not the root of your problem.

[Wylbur]

@bmeeks

This is rather disconcerting for a referbished machine that is less than 30 days old.

I had to put in a second SSD because the system would not install for some reason. So that has me wondering of the referbish didn't detect a bad HDD.

I really hate to pull this right now and run Knoppix to do diagnostics, because it takes me about 30 minute to get the INTEL ethernet adapter out of this box and into the back up unit so I can start that whole process.

Have any thoughts what diagnostics I can run with pfSense in order to capture this?

I had thought this was related to the time change since it happened right abou that time. But, this clock should be GMT/UTC, so only the offset would/should have changed.

Since this box is under warranty, I would like to be able to demonstrate this to the entity where I go it.

Wylbur

[stephenw10]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

Mar 10 09:08:42 kernel hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Mar 10 09:08:42 kernel hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Mar 10 09:08:42 kernel pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Mar 10 09:08:42 kernel pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Mar 10 09:08:42 kernel hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Mar 10 09:08:42 kernel hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Mar 10 09:08:42 kernel pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1

Try disabling all that in the BIOS. And anything else you're not using there. Some of those things could be conflicting with the addon NIC preventing it being detected.

[Sergei_Shablovsky]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

@Sergei_Shablovsky

Thank you for your input. But, I've already done that. This is what the WAN port is running with. The LAN port is whatever the MOBO has and I never seem to have problems with that port. The weirdness is, this MOBO will not accept connections on both the Intel ports of the dual Intel port ethernet adapter that I'm using on this machine. Yet when I ran that adapter in another machine, both ports were usable so one was for WAN and the other for LAN.

But we have to ask this question: Why was I able to run for months on end when using Realtek ports with Untangle, or IPfire with this ISP? That is the thing that puzzels me.

In case everyone ISP use Juniper, Extreme (and other not-so-bad) hardware on aggregate level, and every user use Intel, Melannox (and other bug-free) hardware and well-writed & tested drivers,- we all wouldn’t have any puzzle-problem like this anymore.

So, just “catch, fix and forgot”,- best strategy in this hardware-mixed world. ;)

Now, if I were a "c" (or assembly language) programmer and knew the x86 architecture as well as I do z/Architecture machines (IBM Mainframes), I could probably code a trap to capture this failure and know why it was happening. Or I could run a trace of it that we could examine once it failed. But I don't know this architecture at that level. So unfortunately, I'm more of a knowledgeable user that knows enough to just be dangerous.

Just change the SSD, choose NICs that ISP recommend to work better WITH HIS APPLIANCE, make backups regulary (both config.xml and ZFS snapshots) and be happy until next device upgrade/change.

[Sergei_Shablovsky]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

@keyser

I see no failures that indicate a problem with renewal of lease with the ISP. What I do see are some changes where the fiber optic modem may get its IPv4 IP address changed and then the WAN is given a new IPv4 address. And then some 8.8.8.8 pings take place and some latency is noted.

Why exactly the IP on “fiber optic modem” are changed?
This is very rare situation in fiber nets in Europe, as I know.

What is this device exactly? (Manufacturer and model)

Meanwhile on these latency issues, we know that the ISP has the ability to run Gigabit connections. What we have is 200/200 Mbs.

From which country You are, and ISP ?

[Sergei_Shablovsky]

@keyser said in Back to odd problem -- lose WAN at random points with a week or more between events:

@Wylbur You would know from the logs if renew was failing, because the logs would fill with a lot of renew attempts (with an increasing timer). So thats not the root of your problem.

Agree!!!

[Sergei_Shablovsky]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

@bmeeks

This is rather disconcerting for a referbished machine that is less than 30 days old.

I had to put in a second SSD because the system would not install for some reason. So that has me wondering of the referbish didn't detect a bad HDD.

Since this box is under warranty, I would like to be able to demonstrate this to the entity where I go it.

Save Your time: not spending time on “demonstrations”, return the box, buy something more powerful or from well-known & reputable brand.

[Wylbur]

@stephenw10

I captured a packet trace when I ran into another loss of the system (DHCP working just fine, no ISP access). Unfortuneatley, I lost that text file. The funning thing is, it appeared that there were packets were passing through the WAN. So it seems that something causes communications to fail which is why a reboot clears the issue.

Meanwhile, I have to find a point where I can take the system down, and come back up on the backup machine, while I figure out how to make the BIOS changes. Hopefully this will be simple and not get blocked by the built in security so I can make the changes to the BIOS.

Wylbur.

[Wylbur]

@Sergei_Shablovsky

I am in the USA. The ISP is a company called Metronet. The fiber optic interface system is by Nokia, and is an Intertek unit.

Metronet, Spectrum, AT&T, ComCast, etc. all change your IP address whenever they feel like it so you can't have a static address and host a web site unless you pay them for a static address.

[Sergei_Shablovsky]

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

@Sergei_Shablovsky

I am in the USA. The ISP is a company called Metronet. The fiber optic interface system is by Nokia, and is an Intertek unit.

Ok, thanks. Just to know.

Metronet, Spectrum, AT&T, ComCast, etc. all change your IP address whenever they feel like it so you can't have a static address and host a web site unless you pay them for a static address.

Did DynDNS (or any other services) give You ability to having remote access?

[Wylbur]

@stephenw10

I have swapped systems so that the backup is running and the new system is out for me to change BIOS settings.

So with the new machine that had the log errors below, I do not see any correlation of the following to anything I can change in the BIOS.

Mar 10 09:08:42 kernel hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Mar 10 09:08:42 kernel hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Mar 10 09:08:42 kernel pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Mar 10 09:08:42 kernel pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Mar 10 09:08:42 kernel hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Mar 10 09:08:42 kernel hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Mar 10 09:08:42 kernel pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1

---

I got into the BIOS and did not find anything for changing any of the above.
However, I did find where the system can "sleep" or change to low power for several items. I set all that off.

I also, ran the I/O tests while I had the opportunity on the SSDs and the initial tests came back good. Ran the extended tests and they are also good.

Then I ran the RAM tests and they came out with no errors detected.

The big question I have is, what would cause pfSense to "fail" and stop responding to ping, not respond to its website/page(s), but yet allow an iPhone 5 attached via an adapter that accepts RJ45 (eithernet), and continue streaming data via TuneIn (out of Europe in this case) while not responding to keyboard/mouse attached to the server via USB. Oh, and causing a Roku box to lose its connections (this by Wifi) so that TV(s) so attached lose connections. In otherwords, what makes that iPhone5 special that it did not lose its connections?

And I think this has happened now 3 times.

[stephenw10]

If the firewall is unable to open new states it would present like that. Existing states stay open so traffic continues. I would expect to see that logged though. Especially if it actually ran out of states.

You should be able to disable on-board audio in the BIOS unless it's significantly locked down.