Back to odd problem -- lose WAN at random points with a week or more between events

Wylbur

This log is represntative of what I see when this problem occurs. Note that the LAN is still functional, the WAN is not functional.

This means I can access printers, file server, or any other device inside the LAN

The system is connected to an ISP that uses fiber optics and that interface device is what the WAN is connected to.

BTW I went looking for prior topics I had on this, but couldn't find them.

I thought that the Service_Watchdog package fixed this and it finally happened (again).

We had thought that this was related to the lack of INTEL ethernet chips. Not sure that is the case (currently running using our backup server -- need to replace primary -- that is a whole 'nuther story. This problem has happeed with either server.

I just don't know what to really look at and tips, pointers, I'm just trying to figure it out. [When it happens, I know exactly what to do and I have it all back up as fast as it will boot]

Jan 29 12:17:31 php 99429 [Snort] Snort START for WAN(re1)...
Jan 29 12:17:31 php 99429 [Snort] Snort has restarted on WAN with your new set of rules...
Jan 29 12:17:31 php 99429 [Snort] The Rules update has finished.
Jan 29 15:42:00 sshguard 77838 Exiting on signal.
Jan 29 15:42:00 sshguard 48631 Now monitoring attacks.
Jan 29 16:25:29 rc.gateway_alarm 70811 >>> Gateway alarm: WAN_DHCP (Addr:100.66.96.1 Alarm:1 RTT:2.225ms RTTsd:2.436ms Loss:21%)
Jan 29 16:25:29 check_reload_status 429 updating dyndns WAN_DHCP
Jan 29 16:25:29 check_reload_status 429 Restarting IPsec tunnels
Jan 29 16:25:29 check_reload_status 429 Restarting OpenVPN tunnels/interfaces
Jan 29 16:25:29 check_reload_status 429 Reloading filter
Jan 29 16:25:31 php-fpm 28188 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Jan 29 16:25:31 php-fpm 28188 /rc.openvpn: Gateway, NONE AVAILABLE
Jan 29 16:28:50 php-fpm 55453 /index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Jan 29 16:30:05 php-fpm 54351 /diag_reboot.php: Stopping all packages.
Jan 29 16:30:05 SnortStartup 67805 Snort STOP for WAN(re1)...
Jan 29 16:30:06 snort 22594 *** Caught Term-Signal
Jan 29 16:30:11 kernel re1: promiscuous mode disabled
Jan 29 16:30:14 reroot 91849 rerooted by root
Jan 29 16:30:22 syslogd kernel boot file is /boot/kernel/kernel
Jan 29 16:30:22 kernel pflog0: promiscuous mode disabled
Jan 29 16:30:22 kernel Trying to mount root from zfs:pfSense/ROOT/default []...
Jan 29 16:30:22 kernel CPU: Intel(R) Pentium(R) D CPU 3.40GHz (3391.50-MHz K8-class CPU)
Jan 29 16:30:22 kernel Origin="GenuineIntel" Id=0xf65 Family=0xf Model=0x6 Stepping=5
Jan 29 16:30:22 kernel Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jan 29 16:30:22 kernel Features2=0xe49d<SSE3,DTES64,MON,DS_CPL,EST,CNXT-ID,CX16,xTPR,PDCM>
Jan 29 16:30:22 kernel AMD Features=0x20100800<SYSCALL,NX,LM>
Jan 29 16:30:22 kernel AMD Features2=0x1<LAHF>
Jan 29 16:30:22 kernel TSC: P-state invariant
Jan 29 16:30:22 check_reload_status 65042 rc.newwanip starting re1

stephenw10

The first thing you should do is set the gateway monitoring IP to something external like 8.8.8.8 or 1.1.1.1 to get a better idea of actual connectivity. The ISP gateway, which looks like a CGN address, may not reliably respond to ping. However if you only have one WAN traffic should still try to use it.

It looks like you are using a Realtek NIC (re1) as WAN. Can you reassign it as a non-Realtek NIC?

It's not throwing the usual watchdog error but it could still be failing. However that normally requires a full reboot to recover. You only need to reroot here?

Steve

Wylbur

@stephenw10

I'm currently running with my backup system. I'm going to see if the dual port Intel adapter will fit in it from the Primary. Just need to get some down time for that. Meanwhile, I thought I had it using 8.8.8.8. -- Checked it and, no, I forgot that when I was manually restroring config info. Meanwhile to afirm: Metronet (ISP) is using CGN.

Wylbur

@Wylbur

Forgot to anwer this: Yes, it only takes a reboot, and I typically do a reroot reboot. Don't know that is needed. I have done reboot no reroot. Either seems to solve the issue.

stephenw10

Hmm, well reroot only reloads userland stuff. It does not reload the kernel or drivers or reset hardware. So if that is enough to recover it that implies it isn't a driver/hardware issue.

Wylbur

@stephenw10
Well, it did it again this morning (time stamps in log). I have not had time to check on the INTEL dual port adapter yet... What is the cause? It doesn't happen every time the ISP changes our IP address.

Feb 7 00:10:00 sshguard 18368 Now monitoring attacks.
Feb 7 00:17:10 php 45901 [Snort] Snort Subscriber rules are up to date...
Feb 7 00:17:11 php 45901 [Snort] Snort GPLv2 Community Rules are up to date...
Feb 7 00:17:11 php 45901 [Snort] Emerging Threats Open rules are up to date...
Feb 7 00:17:11 php 45901 [Snort] Feodo Tracker Botnet C2 IP rules file update downloaded successfully.
Feb 7 00:17:11 php 45901 [Snort] Feodo Tracker Botnet C2 IP rules were updated...
Feb 7 00:17:11 php 45901 [Snort] Updating rules configuration for: WAN ...
Feb 7 00:17:13 php 45901 [Snort] Enabling any flowbit-required rules for: WAN...
Feb 7 00:17:13 php 45901 [Snort] Building new sid-msg.map file for WAN...
Feb 7 00:17:14 php 45901 [Snort] Snort STOP for WAN(re1)...
Feb 7 00:17:15 snort 53418 *** Caught Term-Signal
Feb 7 00:17:16 php 45901 [Snort] Snort START for WAN(re1)...
Feb 7 00:17:16 php 45901 [Snort] Snort has restarted on WAN with your new set of rules...
Feb 7 00:17:16 php 45901 [Snort] The Rules update has finished.
Feb 7 00:36:29 rc.gateway_alarm 20913 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.940ms RTTsd:.983ms Loss:22%)
Feb 7 00:36:29 check_reload_status 1079 updating dyndns WAN_DHCP
Feb 7 00:36:29 check_reload_status 1079 Restarting IPsec tunnels
Feb 7 00:36:29 check_reload_status 1079 Restarting OpenVPN tunnels/interfaces
Feb 7 00:36:29 check_reload_status 1079 Reloading filter
Feb 7 00:36:31 php-fpm 85226 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 7 00:36:31 php-fpm 85226 /rc.openvpn: Gateway, NONE AVAILABLE
Feb 7 00:54:00 sshguard 18368 Exiting on signal.
Feb 7 00:54:00 sshguard 44654 Now monitoring attacks.
Feb 7 01:51:00 sshguard 44654 Exiting on signal.
Feb 7 01:51:00 sshguard 17652 Now monitoring attacks.
Feb 7 02:48:00 sshguard 17652 Exiting on signal.
Feb 7 02:48:00 sshguard 82920 Now monitoring attacks.
Feb 7 02:57:00 sshguard 82920 Exiting on signal.
Feb 7 02:57:00 sshguard 39651 Now monitoring attacks.
Feb 7 03:44:00 sshguard 39651 Exiting on signal.
Feb 7 03:44:00 sshguard 30885 Now monitoring attacks.
Feb 7 04:41:00 sshguard 30885 Exiting on signal.
Feb 7 04:41:00 sshguard 19403 Now monitoring attacks.
Feb 7 05:38:00 sshguard 19403 Exiting on signal.
Feb 7 05:38:00 sshguard 1370 Now monitoring attacks.
Feb 7 06:18:27 php 39091 [Snort] Rules download error: Resolving timed out after 10012 milliseconds
Feb 7 06:18:27 php 39091 [Snort] Will retry in 15 seconds...

stephenw10

Ah, do you have more than one IPv4 configured?

If the default ipv4 gateway is set to automatic and you have more than one it may be switching to something invalid.

If you do set the default to WAN_DHCP in System > Routing > Gateways.

Wylbur

@stephenw10

What I have is WAN_DHCP in Gateway showing the globe (current default route as present in the current routing tabel of the O/S.

Default gateway (IPv4) is "Automatic"
Default gateway (IPv6) is "Automatic" --- However IPv6 is otherwise disabled incoming and outgoing.

So it should have picked back up and kept going, right?

And it did it again this AM:

Feb 9 03:11:00 sshguard 48503 Exiting on signal.
Feb 9 03:11:00 sshguard 23891 Now monitoring attacks.
Feb 9 03:36:00 sshguard 23891 Exiting on signal.
Feb 9 03:36:00 sshguard 76069 Now monitoring attacks.
Feb 9 04:01:00 sshguard 76069 Exiting on signal.
Feb 9 04:01:00 sshguard 49322 Now monitoring attacks.
Feb 9 04:05:59 rc.gateway_alarm 28232 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.797ms RTTsd:.947ms Loss:22%)
Feb 9 04:05:59 check_reload_status 429 updating dyndns WAN_DHCP
Feb 9 04:05:59 check_reload_status 429 Restarting IPsec tunnels
Feb 9 04:05:59 check_reload_status 429 Restarting OpenVPN tunnels/interfaces
Feb 9 04:05:59 check_reload_status 429 Reloading filter
Feb 9 04:06:01 php-fpm 90818 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 9 04:06:01 php-fpm 90818 /rc.openvpn: Gateway, NONE AVAILABLE
Feb 9 04:28:00 sshguard 49322 Exiting on signal.

Simple reboot (reroot not done) and we were back up and running. BTW I am up on 2.7.2

It will be sometime this weeking that I will be able to attempt to move the dual port Intel ethernet adapter -- but I will have to find a short "blade" to swap with the full sized one (I've got one here somewhere). So once I find that shorty, I'll attempt to fit it and then swap with the Real Tek that is in the back up system.

Wylbur

@Wylbur
Here is more. What happened is, the ISP was doing maint. Not sure what we lost (routing in general, DNS, or what), but they admitted they were doing maint work. At this point I haven't gotten to the Intel Dual Ethernet adapter swap in. So here is what their maint window looked like to pfSense:

Feb 10 18:17:25 	snort 	14352 	*** Caught Term-Signal
Feb 10 18:17:26 	php 	68322 	[Snort] Snort START for WAN(re1)...
Feb 10 18:17:26 	php 	68322 	[Snort] Snort has restarted on WAN with your new set of rules...
Feb 10 18:17:26 	php 	68322 	[Snort] The Rules update has finished.
Feb 10 19:08:00 	sshguard 	55854 	Exiting on signal.
Feb 10 19:08:00 	sshguard 	4519 	Now monitoring attacks.
Feb 10 19:40:00 	sshguard 	4519 	Exiting on signal.
Feb 10 19:40:00 	sshguard 	94759 	Now monitoring attacks.
Feb 10 20:05:00 	sshguard 	94759 	Exiting on signal.
Feb 10 20:05:00 	sshguard 	45085 	Now monitoring attacks.
Feb 10 21:02:00 	sshguard 	45085 	Exiting on signal.
Feb 10 21:02:00 	sshguard 	33272 	Now monitoring attacks.
Feb 10 21:28:26 	**rc.gateway_alarm** 	18568 	>>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:19.639ms RTTsd:6.562ms Loss:21%)
Feb 10 21:28:26 	check_reload_status 	429 	updating dyndns WAN_DHCP
Feb 10 21:28:26 	check_reload_status 	429 	Restarting IPsec tunnels
Feb 10 21:28:26 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Feb 10 21:28:26 	check_reload_status 	429 	Reloading filter
Feb 10 21:28:28 	php-fpm 	16396 	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 10 21:28:28 	php-fpm 	16396 	/rc.openvpn: Gateway, NONE AVAILABLE
Feb 10 21:29:48 	**rc.gateway_alarm** 	17494 	>>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:0 RTT:17.482ms RTTsd:4.685ms Loss:11%)
Feb 10 21:29:48 	check_reload_status 	429 	updating dyndns WAN_DHCP
Feb 10 21:29:48 	check_reload_status 	429 	Restarting IPsec tunnels
Feb 10 21:29:48 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Feb 10 21:29:48 	check_reload_status 	429 	Reloading filter
Feb 10 21:29:50 	php-fpm 	98627 	/rc.openvpn: Gateway, NONE AVAILABLE
Feb 10 21:29:50 	php-fpm 	98627 	/rc.openvpn: Gateway, NONE AVAILABLE
Feb 10 21:59:00 	sshguard 	33272 	Exiting on signal.
Feb 10 21:59:00 	sshguard 	53623 	Now monitoring attacks.
Feb 10 22:56:00 	sshguard 	53623 	Exiting on signal.
Feb 10 22:56:00 	sshguard 	24897 	Now monitoring attacks.
Feb 10 23:52:00 	sshguard 	24897 	Exiting on signal.
Feb 10 23:52:00 	sshguard 	44085 	Now monitoring attacks.
Feb 11 00:17:10 	php 	3430 	[Snort] Snort Subscriber rules are up to date...
Feb 11 00:17:10 	php 	3430 	[Snort] Snort GPLv2 Community Rules are up to date...
Feb 11 00:17:10 	php 	3430 	[Snort] Emerging Threats Open rules are up to date...
Feb 11 00:17:10 	php 	3430 	[Snort] Feodo Tracker Botnet C2 IP rules file update downloaded successfully.
Feb 11 00:17:10 	php 	3430 	[Snort] Feodo Tracker Botnet C2 IP rules were updated...
Feb 11 00:17:10 	php 	3430 	[Snort] Updating rules configuration for: WAN ...
Feb 11 00:17:13 	php 	3430 	[Snort] Enabling any flowbit-required rules for: WAN...
Feb 11 00:17:13 	php 	3430 	[Snort] Building new sid-msg.map file for WAN...
Feb 11 00:17:13 	php 	3430 	[Snort] Snort STOP for WAN(re1)...
Feb 11 00:17:14 	snort 	96873 	*** Caught Term-Signal
Feb 11 00:17:15 	php 	3430 	[Snort] Snort START for WAN(re1)...
Feb 11 00:17:16 	php 	3430 	[Snort] Snort has restarted on WAN with your new set of rules...
Feb 11 00:17:16 	php 	3430 	[Snort] The Rules update has finished.
Feb 11 00:49:00 	sshguard 	44085 	Exiting on signal.
Feb 11 00:49:00 	sshguard 	7413 	Now monitoring attacks.
Feb 11 01:46:00 	sshguard 	7413 	Exiting on signal.
Feb 11 01:46:00 	sshguard 	8022 	Now monitoring attacks.
Feb 11 02:38:00 	sshguard 	8022 	Exiting on signal.
Feb 11 02:38:00 	sshguard 	93663 	Now monitoring attacks.
Feb 11 02:43:00 	sshguard 	93663 	Exiting on signal.
Feb 11 02:43:00 	sshguard 	88326 	Now monitoring attacks.
Feb 11 03:40:00 	sshguard 	88326 	Exiting on signal.
Feb 11 03:40:00 	sshguard 	39551 	Now monitoring attacks.
Feb 11 04:36:00 	sshguard 	39551 	Exiting on signal.
Feb 11 04:36:00 	sshguard 	97428 	Now monitoring attacks.
Feb 11 05:33:00 	sshguard 	97428 	Exiting on signal.
Feb 11 05:33:00 	sshguard 	11114 	Now monitoring attacks.
Feb 11 06:06:29 	**rc.gateway_alarm** 	21198 	>>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.690ms RTTsd:1.097ms Loss:22%)
Feb 11 06:06:29 	check_reload_status 	429 	updating dyndns WAN_DHCP
Feb 11 06:06:29 	check_reload_status 	429 	Restarting IPsec tunnels
Feb 11 06:06:29 	check_reload_status 	429 	Restarting OpenVPN tunnels/interfaces
Feb 11 06:06:29 	check_reload_status 	429 	Reloading filter
Feb 11 06:06:31 	php-fpm 	52862 	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 11 06:06:31 	php-fpm 	52862 	/rc.openvpn: Gateway, NONE AVAILABLE
Feb 11 06:18:40 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10001 milliseconds
Feb 11 06:18:40 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:20:28 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10010 milliseconds
Feb 11 06:20:28 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:22:15 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10001 milliseconds
Feb 11 06:22:15 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:24:02 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10007 milliseconds
Feb 11 06:24:02 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:24:17 	php 	45146 	File 'snortrules-snapshot-29200.tar.gz.md5' download attempts: 4 ...
Feb 11 06:24:17 	php 	45146 	[Snort] Snort Subscriber rules md5 download failed...
Feb 11 06:24:17 	php 	45146 	[Snort] Server returned error code 0...
Feb 11 06:25:50 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10003 milliseconds
Feb 11 06:25:50 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:27:37 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10008 milliseconds
Feb 11 06:27:37 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:29:24 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10004 milliseconds
Feb 11 06:29:24 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:30:00 	sshguard 	11114 	Exiting on signal.
Feb 11 06:30:00 	sshguard 	15361 	Now monitoring attacks.
Feb 11 06:31:12 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10013 milliseconds
Feb 11 06:31:12 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:31:27 	php 	45146 	File 'community-rules.tar.gz.md5' download attempts: 4 ...
Feb 11 06:31:27 	php 	45146 	[Snort] Snort GPLv2 Community Rules md5 download failed...
Feb 11 06:31:27 	php 	45146 	[Snort] Server returned error code 0...
Feb 11 06:32:59 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10015 milliseconds
Feb 11 06:32:59 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:34:46 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10004 milliseconds
Feb 11 06:34:46 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:36:34 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10009 milliseconds
Feb 11 06:36:34 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:38:21 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10008 milliseconds
Feb 11 06:38:21 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:38:36 	php 	45146 	File 'emerging.rules.tar.gz.md5' download attempts: 4 ...
Feb 11 06:38:36 	php 	45146 	[Snort] Emerging Threats Open rules md5 download failed...
Feb 11 06:38:36 	php 	45146 	[Snort] Server returned error code 0...
Feb 11 06:40:08 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10002 milliseconds
Feb 11 06:40:08 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:41:56 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10001 milliseconds
Feb 11 06:41:56 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:43:43 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10001 milliseconds
Feb 11 06:43:43 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:45:31 	php 	45146 	[Snort] Rules download error: Resolving timed out after 10012 milliseconds
Feb 11 06:45:31 	php 	45146 	[Snort] Will retry in 15 seconds...
Feb 11 06:45:46 	php 	45146 	File 'feodotracker.tar.gz' download attempts: 4 ...
Feb 11 06:45:46 	php 	45146 	[Snort] ERROR: Feodo Tracker Botnet C2 IP rules file download failed... server returned error '0'.
Feb 11 06:45:46 	php 	45146 	[Snort] The Rules update has finished.
Feb 11 07:27:00 	sshguard 	15361 	Exiting on signal.
Feb 11 07:27:00 	sshguard 	13884 	Now monitoring attacks.
Feb 11 08:24:00 	sshguard 	13884 	Exiting on signal.
Feb 11 08:24:00 	sshguard 	8023 	Now monitoring attacks.
Feb 11 08:55:26 	check_reload_status 	429 	rc.newwanip starting re1
Feb 11 08:55:27 	php-fpm 	94729 	/rc.newwanip: rc.newwanip: Info: starting on re1.
**Feb 11 08:55:27** 	php-fpm 	94729 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.97.63) (interface: WAN[wan]) (real interface: re1).
**Feb 11 08:55:27** 	php-fpm 	94729 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 11 08:55:27 	kernel 		[fib_algo] inet.0 (bsearch4#43) rebuild_fd_flm: switching algo to radix4_lockless
Feb 11 08:55:28 	php-fpm 	94729 	/rc.newwanip: Gateway, NONE AVAILABLE
Feb 11 08:55:28 	php-fpm 	94729 	/rc.newwanip: Gateway, NONE AVAILABLE
Feb 11 08:55:28 	php-fpm 	94729 	/rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Feb 11 08:55:28 	php-fpm 	94729 	/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Feb 11 08:55:28 	php-fpm 	94729 	/rc.newwanip: Creating rrd update script
Feb 11 08:55:30 	php-fpm 	94729 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 100.66.97.63 - Restarting packages.
Feb 11 08:55:30 	check_reload_status 	429 	Starting packages
Feb 11 08:55:30 	check_reload_status 	429 	Reloading filter
Feb 11 08:55:31 	php-fpm 	44141 	/rc.start_packages: Restarting/Starting all packages.

Once the Wan (Re1) interface connected, workstations email clients made connections and got email Roku boxes recognized they had internet connections again via WiFI....

stephenw10

I don't see any actual link state changes there. It looks like the WAN just stopped passing traffic and then some time later started again. Twice.

Did you manually reload the WAN at Feb 11 08:55:26?

Wylbur

@stephenw10

No I didn't. And this is under 2.7.2 -- don't know what would have changed with it for this to recover where it didn't before.

But this is goodness, or so it appears. And I didn't get to the adapter swap. Going to find some time this week to do it. The problem is, without our primary being functional, this is a hard outage to our LAN for me to do this work. So I will have to do it some evening after "biz" hours, and online classes.

Wylbur

@Wylbur

Attempted the adapter swap. System would only recognize one port out of two on the adapter. So put WAN on the Intel and the LAN on the MOBO ethernet port.

So this is a backup box again. Got primary system functional. It has its own issue during install.

I have shut down this box and put it on the shelf.

stephenw10

So now waiting to see if the Intel NIC fails in the same way?

Wylbur

@stephenw10

Yes, with the current box. And I have it running these:
Avahi
dpinger
snort
zeek

And it seems to be running quite well so far.

Wylbur

@Wylbur WAN failure has happened twice now. This log is from the second Failure. [I'm going throught packages to see if I missed some setting... Only thing I can think of since WAN is on Intel ethernet chipset...)

Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Feb 23 08:44:57 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 23 08:44:58 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Feb 23 08:44:58 	kernel 		done.
Feb 23 08:44:58 	kernel 		pflog0: promiscuous mode enabled
Feb 23 08:44:58 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 23 08:44:58 	kernel 		.done.
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	php-cgi 	508 	rc.bootup: sync unbound done.
Feb 23 08:44:59 	kernel 		done.
Feb 23 08:44:59 	check_reload_status 	473 	Linkup starting em0
Feb 23 08:44:59 	kernel 		em0: link state changed to UP
Feb 23 08:45:00 	php-fpm 	410 	/rc.linkup: Ignoring link event during boot sequence.
Feb 23 08:45:06 	kernel 		done.
Feb 23 08:45:06 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Feb 23 08:45:06 	kernel 		done.
Feb 23 08:45:07 	kernel 		done.
Feb 23 08:45:08 	kernel 		done.
Feb 23 08:45:08 	check_reload_status 	473 	Updating all dyndns
Feb 23 08:45:08 	kernel 		....
Feb 23 08:45:08 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Feb 23 08:45:08 	kernel 		.done.
Feb 23 08:45:12 	php-cgi 	508 	rc.bootup: Creating rrd update script
Feb 23 08:45:12 	syslogd 		exiting on signal 15
Feb 23 08:45:13 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 23 08:45:13 	kernel 		done.
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: Restarting/Starting all packages.
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: Starting service avahi
Feb 23 08:45:13 	php-fpm 	410 	/rc.start_packages: [zeek] Starting service...
Feb 23 08:45:13 	SnortStartup 	73288 	Snort START for WAN(igb1)...
Feb 23 08:45:13 	kernel 		igb1: promiscuous mode enabled
Feb 23 08:45:13 	avahi-daemon 	70713 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 23 08:45:13 	avahi-daemon 	70713 	Successfully dropped root privileges.
Feb 23 08:45:13 	avahi-daemon 	70713 	avahi-daemon 0.8 starting up.
Feb 23 08:45:13 	avahi-daemon 	70713 	No service file found in /usr/local/etc/avahi/services.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface lo0.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface em0.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Feb 23 08:45:13 	avahi-daemon 	70713 	New relevant interface igb1.IPv4 for mDNS.
Feb 23 08:45:13 	avahi-daemon 	70713 	Network interface enumeration completed.
Feb 23 08:45:13 	avahi-daemon 	70713 	Server startup complete. Host name is pfSense.local. Local service cookie is 2406046792.
Feb 23 08:45:16 	php-fpm 	410 	/rc.start_packages: [zeek] Adding cronjobs ...
Feb 23 08:45:16 	root 	96059 	Bootup complete
Feb 23 08:45:17 	login 	96637 	login on ttyv0 as root
Feb 23 08:45:17 	sshguard 	98598 	Now monitoring attacks.
Feb 23 08:45:34 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 23 10:17:00 	sshguard 	98598 	Exiting on signal.
Feb 23 10:17:00 	sshguard 	81780 	Now monitoring attacks.
Feb 23 18:45:00 	sshguard 	81780 	Exiting on signal.
Feb 23 18:45:00 	sshguard 	1738 	Now monitoring attacks.
Feb 23 20:38:00 	sshguard 	1738 	Exiting on signal.
Feb 23 20:38:00 	sshguard 	12967 	Now monitoring attacks.
Feb 24 01:46:00 	sshguard 	12967 	Exiting on signal.
Feb 24 01:46:00 	sshguard 	56276 	Now monitoring attacks.
Feb 24 08:35:00 	sshguard 	56276 	Exiting on signal.
Feb 24 08:35:00 	sshguard 	199 	Now monitoring attacks.
Feb 24 12:22:00 	sshguard 	199 	Exiting on signal.
Feb 24 12:22:00 	sshguard 	56952 	Now monitoring attacks.
Feb 24 21:00:00 	sshguard 	56952 	Exiting on signal.
Feb 24 21:00:00 	sshguard 	61484 	Now monitoring attacks.
Feb 24 21:44:59 	rc.gateway_alarm 	76930 	>>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.974ms RTTsd:.992ms Loss:21%)
Feb 24 21:44:59 	check_reload_status 	473 	updating dyndns WAN_DHCP
Feb 24 21:44:59 	check_reload_status 	473 	Restarting IPsec tunnels
Feb 24 21:44:59 	check_reload_status 	473 	Restarting OpenVPN tunnels/interfaces
Feb 24 21:44:59 	check_reload_status 	473 	Reloading filter
Feb 24 21:45:00 	sshguard 	61484 	Exiting on signal.
Feb 24 21:45:00 	sshguard 	80832 	Now monitoring attacks.
Feb 24 21:45:00 	php-fpm 	409 	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 24 21:45:00 	php-fpm 	409 	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 24 21:57:22 	php-fpm 	409 	/index.php: Session timed out for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:24 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:55 	php-fpm 	410 	/diag_reboot.php: Stopping all packages.
Feb 24 21:57:55 	SnortStartup 	72115 	Snort STOP for WAN(igb1)...
Feb 24 21:57:56 	snort 	74401 	*** Caught Term-Signal
Feb 24 21:57:59 	kernel 		igb1: promiscuous mode disabled
Feb 24 21:58:00 	php-cgi 	37048 	servicewatchdog_cron.php: Service Watchdog detected service zeek stopped. Restarting zeek (Zeek Network Security Monitor service)
Feb 24 21:58:00 	php-cgi 	37048 	servicewatchdog_cron.php: Service Watchdog detected service snort stopped. Restarting snort (Snort IDS/IPS Daemon)
Feb 24 21:58:00 	SnortStartup 	41935 	Snort START for WAN(igb1)...
Feb 24 21:58:00 	kernel 		igb1: promiscuous mode enabled
Feb 24 21:58:02 	reboot 	36197 	rebooted by root
Feb 24 21:58:02 	syslogd 		exiting on signal 15
Feb 24 21:58:55 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 24 21:58:55 	kernel 		pflog0: promiscuous mode disabled
Feb 24 21:58:55 	kernel 		igb1: promiscuous mode disabled
Feb 24 21:58:55 	kernel 		Waiting (max 60 seconds) for system process `vnlru' to stop... done
Feb 24 21:58:55 	kernel 		Waiting (max 60 seconds) for system process `syncer' to stop...
Feb 24 21:58:55 	kernel 		Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
Feb 24 21:58:55 	kernel 		All buffers synced.
Feb 24 21:58:55 	kernel 		Uptime: 1d13h13m42s
Feb 24 21:58:55 	kernel 		---<<BOOT>>---
Feb 24 21:58:55 	kernel 		Copyright (c) 1992-2023 The FreeBSD Project.
Feb 24 21:58:55 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Feb 24 21:58:55 	kernel 		The Regents of the University of California. All rights reserved.
Feb 24 21:58:55 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Feb 24 21:58:55 	kernel 		FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023
Feb 24 21:58:55 	kernel 		root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64
Feb 24 21:58:55 	kernel 		FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Feb 24 21:58:55 	kernel 		VT(efifb): resolution 1024x768
Feb 24 21:58:55 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3200.00-MHz K8-class CPU)
Feb 24 21:58:55 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Feb 24 21:58:55 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Feb 24 21:58:55 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Feb 24 21:58:55 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Feb 24 21:58:55 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Feb 24 21:58:55 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Feb 24 21:58:55 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Feb 24 21:58:55 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Feb 24 21:58:55 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Feb 24 21:58:55 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Feb 24 21:58:55 	kernel 		TSC: P-state invariant, performance statistics
Feb 24 21:58:55 	kernel 		real memory = 17179869184 (16384 MB)
Feb 24 21:58:55 	kernel 		avail memory = 16475234304 (15712 MB)
Feb 24 21:58:55 	kernel 		Event timer "LAPIC" quality 600
Feb 24 21:58:55 	kernel 		ACPI APIC Table: <HPQOEM 805D >
Feb 24 21:58:55 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Feb 24 21:58:55 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
Feb 24 21:58:55 	kernel 		random: registering fast source Intel Secure Key RNG
Feb 24 21:58:55 	kernel 		random: fast provider: "Intel Secure Key RNG"
Feb 24 21:58:55 	kernel 		random: unblocking device.
Feb 24 21:58:55 	kernel 		ioapic0 <Version 2.0> irqs 0-119
Feb 24 21:58:55 	kernel 		Launching APs: 2 1 3
Feb 24 21:58:55 	kernel 		TCP_ratelimit: Is now initialized
Feb 24 21:58:55 	kernel 		wlan: mac acl policy registered
Feb 24 21:58:55 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1
Feb 24 21:58:55 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
Feb 24 21:58:55 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Feb 24 21:58:55 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1
Feb 24 21:58:55 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Feb 24 21:58:55 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Feb 24 21:58:55 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1
Feb 24 21:58:55 	kernel 		random: entropy device external interface
Feb 24 21:58:55 	kernel 		kbd1 at kbdmux0
Feb 24 21:58:55 	kernel 		WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
Feb 24 21:58:55 	kernel 		netgate0: <unknown hardware>
Feb 24 21:58:55 	kernel 		efirtc0: <EFI Realtime Clock>
Feb 24 21:58:55 	kernel 		efirtc0: registered as a time-of-day clock, resolution 1.000000s
Feb 24 21:58:55 	kernel 		smbios0: <System Management BIOS> at iomem 0xc97cb000-0xc97cb01e
Feb 24 21:58:55 	kernel 		smbios0: Version: 2.7, BCD Revision: 2.7
Feb 24 21:58:55 	kernel 		acpi0: <HPQOEM SLIC-BPC>
Feb 24 21:58:55 	kernel 		acpi0: Power Button (fixed)
Feb 24 21:58:55 	kernel 		cpu0: <ACPI CPU> on acpi0
Feb 24 21:58:55 	kernel 		hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Feb 24 21:58:55 	kernel 		Timecounter "HPET" frequency 24000000 Hz quality 950
Feb 24 21:58:55 	kernel 		Event timer "HPET" frequency 24000000 Hz quality 550
Feb 24 21:58:55 	kernel 		Event timer "HPET1" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET2" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET3" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		Event timer "HPET4" frequency 24000000 Hz quality 440
Feb 24 21:58:55 	kernel 		atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Feb 24 21:58:55 	kernel 		atrtc0: Warning: Couldn't map I/O.
Feb 24 21:58:55 	kernel 		atrtc0: registered as a time-of-day clock, resolution 1.000000s
Feb 24 21:58:55 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
Feb 24 21:58:55 	kernel 		attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Feb 24 21:58:55 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
Feb 24 21:58:55 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
Feb 24 21:58:55 	kernel 		Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Feb 24 21:58:55 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
Feb 24 21:58:55 	kernel 		acpi_ec0: <Embedded Controller: GPE 0x6e> port 0x62,0x66 on acpi0
Feb 24 21:58:55 	kernel 		pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Feb 24 21:58:55 	kernel 		pci0: <ACPI PCI bus> on pcib0
Feb 24 21:58:55 	kernel 		vgapci0: <VGA-compatible display> port 0x4000-0x403f mem 0xe0000000-0xe0ffffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
Feb 24 21:58:55 	kernel 		vgapci0: Boot video device
Feb 24 21:58:55 	kernel 		xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xe1320000-0xe132ffff at device 20.0 on pci0
Feb 24 21:58:55 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
Feb 24 21:58:55 	kernel 		usbus0 on xhci0
Feb 24 21:58:55 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
Feb 24 21:58:55 	kernel 		pci0: <simple comms> at device 22.0 (no driver attached)
Feb 24 21:58:55 	kernel 		uart2: <100 Series/C230 Series Chipset Family KT Redirection> port 0x4080-0x4087 mem 0xe134a000-0xe134afff at device 22.3 on pci0
Feb 24 21:58:55 	kernel 		ahci0: <Intel Sunrise Point AHCI SATA controller> port 0x4088-0x408f,0x4090-0x4093,0x4040-0x405f mem 0xe1348000-0xe1349fff,0xe134f000-0xe134f0ff,0xe134d000-0xe134d7ff at device 23.0 on pci0
Feb 24 21:58:55 	kernel 		ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
Feb 24 21:58:55 	kernel 		ahcich0: <AHCI channel> at channel 0 on ahci0
Feb 24 21:58:55 	kernel 		ahcich1: <AHCI channel> at channel 1 on ahci0
Feb 24 21:58:55 	kernel 		ahcich2: <AHCI channel> at channel 2 on ahci0
Feb 24 21:58:55 	kernel 		ahcich3: <AHCI channel> at channel 3 on ahci0
Feb 24 21:58:55 	kernel 		ahcich4: <AHCI channel> at channel 4 on ahci0
Feb 24 21:58:55 	kernel 		ahcich5: <AHCI channel> at channel 5 on ahci0
Feb 24 21:58:55 	kernel 		ahciem0: <AHCI enclosure management bridge> on ahci0
Feb 24 21:58:55 	kernel 		pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
Feb 24 21:58:55 	kernel 		pci1: <ACPI PCI bus> on pcib1
Feb 24 21:58:55 	kernel 		igb0: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3000-0x301f mem 0xe1200000-0xe121ffff,0xe1000000-0xe10fffff,0xe1240000-0xe1243fff at device 0.0 on pci1
Feb 24 21:58:55 	kernel 		igb0: EEPROM V1.13-1
Feb 24 21:58:55 	kernel 		igb0: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		igb0: Using 4 RX queues 4 TX queues
Feb 24 21:58:55 	kernel 		igb0: Using MSI-X interrupts with 5 vectors
Feb 24 21:58:55 	kernel 		igb0: Ethernet address: 00:1b:21:27:12:36
Feb 24 21:58:55 	kernel 		igb0: netmap queues/slots: TX 4/1024, RX 4/1024
Feb 24 21:58:55 	kernel 		igb1: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3020-0x303f mem 0xe1220000-0xe123ffff,0xe1100000-0xe11fffff,0xe1244000-0xe1247fff at device 0.1 on pci1
Feb 24 21:58:55 	kernel 		igb1: EEPROM V1.13-1
Feb 24 21:58:55 	kernel 		igb1: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		igb1: Using 4 RX queues 4 TX queues
Feb 24 21:58:55 	kernel 		igb1: Using MSI-X interrupts with 5 vectors
Feb 24 21:58:55 	kernel 		igb1: Ethernet address: 00:1b:21:27:12:37
Feb 24 21:58:55 	kernel 		igb1: netmap queues/slots: TX 4/1024, RX 4/1024
Feb 24 21:58:55 	kernel 		isab0: <PCI-ISA bridge> at device 31.0 on pci0
Feb 24 21:58:55 	kernel 		isa0: <ISA bus> on isab0
Feb 24 21:58:55 	kernel 		pci0: <memory> at device 31.2 (no driver attached)
Feb 24 21:58:55 	kernel 		hdac0: <Intel Sunrise Point HDA Controller> mem 0xe1344000-0xe1347fff,0xe1330000-0xe133ffff at device 31.3 on pci0
Feb 24 21:58:55 	kernel 		em0: <Intel(R) I219-LM SPT-H(2)> mem 0xe1300000-0xe131ffff at device 31.6 on pci0
Feb 24 21:58:55 	kernel 		em0: EEPROM V0.8-4
Feb 24 21:58:55 	kernel 		em0: Using 1024 TX descriptors and 1024 RX descriptors
Feb 24 21:58:55 	kernel 		em0: Using an MSI interrupt
Feb 24 21:58:55 	kernel 		em0: Ethernet address: 70:5a:0f:3b:19:12
Feb 24 21:58:55 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
Feb 24 21:58:55 	kernel 		acpi_button0: <Sleep Button> on acpi0
Feb 24 21:58:55 	kernel 		acpi_button1: <Power Button> on acpi0
Feb 24 21:58:55 	kernel 		atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Feb 24 21:58:55 	kernel 		atkbd0: <AT Keyboard> irq 1 on atkbdc0
Feb 24 21:58:55 	kernel 		kbd0 at atkbd0
Feb 24 21:58:55 	kernel 		atkbd0: [GIANT-LOCKED]
Feb 24 21:58:55 	kernel 		acpi_syscontainer0: <System Container> on acpi0
Feb 24 21:58:55 	kernel 		uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Feb 24 21:58:55 	kernel 		hwpstate_intel0: <Intel Speed Shift> on cpu0
Feb 24 21:58:55 	kernel 		hwpstate_intel1: <Intel Speed Shift> on cpu1
Feb 24 21:58:55 	kernel 		hwpstate_intel2: <Intel Speed Shift> on cpu2
Feb 24 21:58:55 	kernel 		hwpstate_intel3: <Intel Speed Shift> on cpu3
Feb 24 21:58:55 	kernel 		Timecounter "TSC-low" frequency 1595996789 Hz quality 1000
Feb 24 21:58:55 	kernel 		Timecounters tick every 1.000 msec
Feb 24 21:58:55 	kernel 		ZFS filesystem version: 5
Feb 24 21:58:55 	kernel 		ZFS storage pool version: features support (5000)
Feb 24 21:58:55 	kernel 		hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Feb 24 21:58:55 	kernel 		hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Feb 24 21:58:55 	kernel 		pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Feb 24 21:58:55 	kernel 		pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Feb 24 21:58:55 	kernel 		hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Feb 24 21:58:55 	kernel 		hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Feb 24 21:58:55 	kernel 		pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1
Feb 24 21:58:55 	kernel 		Trying to mount root from zfs:pfSense/ROOT/default []...
Feb 24 21:58:55 	kernel 		ugen0.1: <Intel XHCI root HUB> at usbus0
Feb 24 21:58:55 	kernel 		uhub0 on usbus0
Feb 24 21:58:55 	kernel 		uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Feb 24 21:58:55 	kernel 		Root mount waiting for: usbus0 CAM
Feb 24 21:58:55 	kernel 		uhub0: 24 ports with 24 removable, self powered
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		Root mount waiting for: CAM
Feb 24 21:58:55 	kernel 		ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
Feb 24 21:58:55 	kernel 		ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Feb 24 21:58:55 	kernel 		ses0: SEMB SES Device
Feb 24 21:58:55 	kernel 		ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Feb 24 21:58:55 	kernel 		ada0: <KingFast HP2818E3> ACS-2 ATA SATA 3.x device
Feb 24 21:58:55 	kernel 		ada0: Serial Number CI0311M000805
Feb 24 21:58:55 	kernel 		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 4096bytes)
Feb 24 21:58:55 	kernel 		ada0: Command Queueing enabled
Feb 24 21:58:55 	kernel 		ada0: 244198MB (500118192 512 byte sectors)
Feb 24 21:58:55 	kernel 		ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Feb 24 21:58:55 	kernel 		ada1: <SATA SSD SBFM61.5> ACS-4 ATA SATA 3.x device
Feb 24 21:58:55 	kernel 		ada1: Serial Number 23052925600661
Feb 24 21:58:55 	kernel 		ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Feb 24 21:58:55 	kernel 		ada1: Command Queueing enabled
Feb 24 21:58:55 	kernel 		ada1: 244198MB (500118192 512 byte sectors)
Feb 24 21:58:55 	kernel 		ses0: pass0,ada0 in 'Slot 00', SATA Slot: scbus0 target 0
Feb 24 21:58:55 	kernel 		ses0: pass1,ada1 in 'Slot 01', SATA Slot: scbus1 target 0
Feb 24 21:58:55 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
Feb 24 21:58:55 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Feb 24 21:58:55 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Feb 24 21:58:55 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Feb 24 21:58:55 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Feb 24 21:58:55 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Feb 24 21:58:55 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Feb 24 21:58:55 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Feb 24 21:58:55 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Feb 24 21:58:55 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Feb 24 21:58:55 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Feb 24 21:58:55 	kernel 		TSC: P-state invariant, performance statistics
Feb 24 21:58:55 	kernel 		pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xe134b000-0xe134bfff at device 20.2 on pci0
Feb 24 21:58:55 	kernel 		ichsmb0: <Intel Sunrise Point-H SMBus controller> port 0xefa0-0xefbf mem 0xe134e000-0xe134e0ff at device 31.4 on pci0
Feb 24 21:58:55 	kernel 		smbus0: <System Management Bus> on ichsmb0
Feb 24 21:58:55 	kernel 		lo0: link state changed to UP
Feb 24 21:58:58 	check_reload_status 	473 	Linkup starting igb1
Feb 24 21:58:58 	kernel 		igb1: link state changed to UP
Feb 24 21:59:02 	check_reload_status 	473 	rc.newwanip starting igb1
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Feb 24 21:59:03 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 24 21:59:04 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Feb 24 21:59:04 	kernel 		done.
Feb 24 21:59:04 	kernel 		pflog0: promiscuous mode enabled
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Feb 24 21:59:05 	kernel 		.done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:05 	php-cgi 	508 	rc.bootup: sync unbound done.
Feb 24 21:59:05 	kernel 		done.
Feb 24 21:59:06 	check_reload_status 	473 	Linkup starting em0
Feb 24 21:59:06 	kernel 		em0: link state changed to UP
Feb 24 21:59:07 	php-fpm 	410 	/rc.linkup: Ignoring link event during boot sequence.
Feb 24 21:59:11 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Feb 24 21:59:11 	kernel 		done.
Feb 24 21:59:12 	kernel 		done.
Feb 24 21:59:12 	kernel 		done.
Feb 24 21:59:12 	check_reload_status 	473 	Updating all dyndns
Feb 24 21:59:13 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Feb 24 21:59:13 	kernel 		.done.
Feb 24 21:59:17 	php-cgi 	508 	rc.bootup: Creating rrd update script
Feb 24 21:59:17 	kernel 		done.
Feb 24 21:59:17 	syslogd 		exiting on signal 15
Feb 24 21:59:17 	syslogd 		kernel boot file is /boot/kernel/kernel
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: Restarting/Starting all packages.
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: Starting service avahi
Feb 24 21:59:17 	php-fpm 	410 	/rc.start_packages: [zeek] Starting service...
Feb 24 21:59:17 	SnortStartup 	58538 	Snort START for WAN(igb1)...
Feb 24 21:59:17 	avahi-daemon 	55968 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Feb 24 21:59:17 	kernel 		igb1: promiscuous mode enabled
Feb 24 21:59:17 	avahi-daemon 	55968 	Successfully dropped root privileges.
Feb 24 21:59:17 	avahi-daemon 	55968 	avahi-daemon 0.8 starting up.
Feb 24 21:59:17 	avahi-daemon 	55968 	No service file found in /usr/local/etc/avahi/services.
Feb 24 21:59:17 	avahi-daemon 	55968 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Feb 24 21:59:17 	avahi-daemon 	55968 	New relevant interface lo0.IPv4 for mDNS.
Feb 24 21:59:17 	avahi-daemon 	55968 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Feb 24 21:59:18 	avahi-daemon 	55968 	New relevant interface em0.IPv4 for mDNS.
Feb 24 21:59:18 	avahi-daemon 	55968 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Feb 24 21:59:18 	avahi-daemon 	55968 	New relevant interface igb1.IPv4 for mDNS.
Feb 24 21:59:18 	avahi-daemon 	55968 	Network interface enumeration completed.
Feb 24 21:59:18 	avahi-daemon 	55968 	Server startup complete. Host name is pfSense.local. Local service cookie is 490430812.
Feb 24 21:59:20 	php-fpm 	410 	/rc.start_packages: [zeek] Adding cronjobs ...
Feb 24 21:59:20 	root 	80968 	Bootup complete
Feb 24 21:59:21 	login 	91105 	login on ttyv0 as root
Feb 24 21:59:21 	sshguard 	92617 	Now monitoring attacks.
Feb 24 22:00:44 	php-fpm 	409 	/index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)

stephenw10

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

Feb 24 21:00:00 sshguard 61484 Now monitoring attacks.
Feb 24 21:44:59 rc.gateway_alarm 76930 >>> Gateway alarm: WAN_DHCP (Addr:8.8.8.8 Alarm:1 RTT:15.974ms RTTsd:.992ms Loss:21%)
Feb 24 21:44:59 check_reload_status 473 updating dyndns WAN_DHCP
Feb 24 21:44:59 check_reload_status 473 Restarting IPsec tunnels
Feb 24 21:44:59 check_reload_status 473 Restarting OpenVPN tunnels/interfaces
Feb 24 21:44:59 check_reload_status 473 Reloading filter
Feb 24 21:45:00 sshguard 61484 Exiting on signal.
Feb 24 21:45:00 sshguard 80832 Now monitoring attacks.
Feb 24 21:45:00 php-fpm 409 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 24 21:45:00 php-fpm 409 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 24 21:57:22 php-fpm 409 /index.php: Session timed out for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:24 php-fpm 409 /index.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)
Feb 24 21:57:55 php-fpm 410 /diag_reboot.php: Stopping all packages.

Ok so it doesn't lose link on WAN. It looks like the WAN simply stops passing traffic.

I would check the ARP table when the WAN fails, see if the gateway still shows.

Then run a packet capture on the WAN interface and see what's actually happening at the time.

Steve

Wylbur

@stephenw10

New symptoms. To get us on the same page, this is the main pfSense server (new built system). This hardware/system was used for the last report.

"Error 5, Unretryable error" I've not seen this before...... [copied this out of the log that is below] --- could this be related to time change?

Meanwhile, I had an iPhone streaming TuneIn from Ireland (from yesterday afternoon) when the following happened and it was still streaming until I forced a power down for reboot of the pfSense server (I could not log into it, ping of the 192.168.1.1 (pfSense) failed, which explains why I couldn't login as admin to issue commands). I didn't think to use a wired keyboard & mouse -- was not quite awake. So I also couldn't get a packet trace or reboot before doing a forced power down and restart.

This is what I captured in the logs once I got the system rebooted:

Mar 9 06:38:00 	sshguard 	44643 	Exiting on signal.
Mar 9 06:38:00 	sshguard 	92365 	Now monitoring attacks.
Mar 9 17:46:00 	sshguard 	92365 	Exiting on signal.
Mar 9 17:46:00 	sshguard 	4750 	Now monitoring attacks.
Mar 9 18:45:00 	sshguard 	4750 	Exiting on signal.
Mar 9 18:45:00 	sshguard 	15612 	Now monitoring attacks.
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 38 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 f8 17 e9 40 08 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 28 e0 19 ab 40 0a 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2a 28 40 00 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2c cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2e cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 30 40 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 70 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 	kernel 		(ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 09:08:42 	syslogd 		kernel boot file is /boot/kernel/kernel
Mar 10 09:08:42 	kernel 		---<<BOOT>>---
Mar 10 09:08:42 	kernel 		Copyright (c) 1992-2023 The FreeBSD Project.
Mar 10 09:08:42 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Mar 10 09:08:42 	kernel 		The Regents of the University of California. All rights reserved.
Mar 10 09:08:42 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Mar 10 09:08:42 	kernel 		FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec 6 20:45:47 UTC 2023
Mar 10 09:08:42 	kernel 		root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/amd64.amd64/sys/pfSense amd64
Mar 10 09:08:42 	kernel 		FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Mar 10 09:08:42 	kernel 		VT(efifb): resolution 1024x768
Mar 10 09:08:42 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3200.00-MHz K8-class CPU)
Mar 10 09:08:42 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Mar 10 09:08:42 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Mar 10 09:08:42 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Mar 10 09:08:42 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Mar 10 09:08:42 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Mar 10 09:08:42 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Mar 10 09:08:42 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Mar 10 09:08:42 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Mar 10 09:08:42 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Mar 10 09:08:42 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Mar 10 09:08:42 	kernel 		TSC: P-state invariant, performance statistics
Mar 10 09:08:42 	kernel 		real memory = 17179869184 (16384 MB)
Mar 10 09:08:42 	kernel 		avail memory = 16475234304 (15712 MB)
Mar 10 09:08:42 	kernel 		Event timer "LAPIC" quality 600
Mar 10 09:08:42 	kernel 		ACPI APIC Table: <HPQOEM 805D >
Mar 10 09:08:42 	kernel 		FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Mar 10 09:08:42 	kernel 		FreeBSD/SMP: 1 package(s) x 4 core(s)
Mar 10 09:08:42 	kernel 		random: registering fast source Intel Secure Key RNG
Mar 10 09:08:42 	kernel 		random: fast provider: "Intel Secure Key RNG"
Mar 10 09:08:42 	kernel 		random: unblocking device.
Mar 10 09:08:42 	kernel 		ioapic0 <Version 2.0> irqs 0-119
Mar 10 09:08:42 	kernel 		Launching APs: 3 2 1
Mar 10 09:08:42 	kernel 		TCP_ratelimit: Is now initialized
Mar 10 09:08:42 	kernel 		wlan: mac acl policy registered
Mar 10 09:08:42 	kernel 		ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff807475a0, 0) error 1
Mar 10 09:08:42 	kernel 		ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80747650, 0) error 1
Mar 10 09:08:42 	kernel 		ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Mar 10 09:08:42 	kernel 		ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80747700, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80765180, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80765230, 0) error 1
Mar 10 09:08:42 	kernel 		iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
Mar 10 09:08:42 	kernel 		iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Mar 10 09:08:42 	kernel 		module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff807652e0, 0) error 1
Mar 10 09:08:42 	kernel 		random: entropy device external interface
Mar 10 09:08:42 	kernel 		kbd1 at kbdmux0
Mar 10 09:08:42 	kernel 		WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
Mar 10 09:08:42 	kernel 		netgate0: <unknown hardware>
Mar 10 09:08:42 	kernel 		efirtc0: <EFI Realtime Clock>
Mar 10 09:08:42 	kernel 		efirtc0: registered as a time-of-day clock, resolution 1.000000s
Mar 10 09:08:42 	kernel 		smbios0: <System Management BIOS> at iomem 0xc97cb000-0xc97cb01e
Mar 10 09:08:42 	kernel 		smbios0: Version: 2.7, BCD Revision: 2.7
Mar 10 09:08:42 	kernel 		acpi0: <HPQOEM SLIC-BPC>
Mar 10 09:08:42 	kernel 		acpi0: Power Button (fixed)
Mar 10 09:08:42 	kernel 		cpu0: <ACPI CPU> on acpi0
Mar 10 09:08:42 	kernel 		hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Mar 10 09:08:42 	kernel 		Timecounter "HPET" frequency 24000000 Hz quality 950
Mar 10 09:08:42 	kernel 		Event timer "HPET" frequency 24000000 Hz quality 550
Mar 10 09:08:42 	kernel 		Event timer "HPET1" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET2" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET3" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		Event timer "HPET4" frequency 24000000 Hz quality 440
Mar 10 09:08:42 	kernel 		atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Mar 10 09:08:42 	kernel 		atrtc0: Warning: Couldn't map I/O.
Mar 10 09:08:42 	kernel 		atrtc0: registered as a time-of-day clock, resolution 1.000000s
Mar 10 09:08:42 	kernel 		Event timer "RTC" frequency 32768 Hz quality 0
Mar 10 09:08:42 	kernel 		attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Mar 10 09:08:42 	kernel 		Timecounter "i8254" frequency 1193182 Hz quality 0
Mar 10 09:08:42 	kernel 		Event timer "i8254" frequency 1193182 Hz quality 100
Mar 10 09:08:42 	kernel 		Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Mar 10 09:08:42 	kernel 		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
Mar 10 09:08:42 	kernel 		acpi_ec0: <Embedded Controller: GPE 0x6e> port 0x62,0x66 on acpi0
Mar 10 09:08:42 	kernel 		pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Mar 10 09:08:42 	kernel 		pci0: <ACPI PCI bus> on pcib0
Mar 10 09:08:42 	kernel 		vgapci0: <VGA-compatible display> port 0x4000-0x403f mem 0xe0000000-0xe0ffffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
Mar 10 09:08:42 	kernel 		vgapci0: Boot video device
Mar 10 09:08:42 	kernel 		xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xe1320000-0xe132ffff at device 20.0 on pci0
Mar 10 09:08:42 	kernel 		xhci0: 32 bytes context size, 64-bit DMA
Mar 10 09:08:42 	kernel 		usbus0 on xhci0
Mar 10 09:08:42 	kernel 		usbus0: 5.0Gbps Super Speed USB v3.0
Mar 10 09:08:42 	kernel 		pci0: <simple comms> at device 22.0 (no driver attached)
Mar 10 09:08:42 	kernel 		uart2: <100 Series/C230 Series Chipset Family KT Redirection> port 0x4080-0x4087 mem 0xe134a000-0xe134afff at device 22.3 on pci0
Mar 10 09:08:42 	kernel 		ahci0: <Intel Sunrise Point AHCI SATA controller> port 0x4088-0x408f,0x4090-0x4093,0x4040-0x405f mem 0xe1348000-0xe1349fff,0xe134f000-0xe134f0ff,0xe134d000-0xe134d7ff at device 23.0 on pci0
Mar 10 09:08:42 	kernel 		ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
Mar 10 09:08:42 	kernel 		ahcich0: <AHCI channel> at channel 0 on ahci0
Mar 10 09:08:42 	kernel 		ahcich1: <AHCI channel> at channel 1 on ahci0
Mar 10 09:08:42 	kernel 		ahcich2: <AHCI channel> at channel 2 on ahci0
Mar 10 09:08:42 	kernel 		ahcich3: <AHCI channel> at channel 3 on ahci0
Mar 10 09:08:42 	kernel 		ahcich4: <AHCI channel> at channel 4 on ahci0
Mar 10 09:08:42 	kernel 		ahcich5: <AHCI channel> at channel 5 on ahci0
Mar 10 09:08:42 	kernel 		ahciem0: <AHCI enclosure management bridge> on ahci0
Mar 10 09:08:42 	kernel 		pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
Mar 10 09:08:42 	kernel 		pci1: <ACPI PCI bus> on pcib1
Mar 10 09:08:42 	kernel 		igb0: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3000-0x301f mem 0xe1200000-0xe121ffff,0xe1000000-0xe10fffff,0xe1240000-0xe1243fff at device 0.0 on pci1
Mar 10 09:08:42 	kernel 		igb0: EEPROM V1.13-1
Mar 10 09:08:42 	kernel 		igb0: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		igb0: Using 4 RX queues 4 TX queues
Mar 10 09:08:42 	kernel 		igb0: Using MSI-X interrupts with 5 vectors
Mar 10 09:08:42 	kernel 		igb0: Ethernet address: 00:1b:21:27:12:36
Mar 10 09:08:42 	kernel 		igb0: netmap queues/slots: TX 4/1024, RX 4/1024
Mar 10 09:08:42 	kernel 		igb1: <Intel(R) PRO/1000 82575EB (Copper)> port 0x3020-0x303f mem 0xe1220000-0xe123ffff,0xe1100000-0xe11fffff,0xe1244000-0xe1247fff at device 0.1 on pci1
Mar 10 09:08:42 	kernel 		igb1: EEPROM V1.13-1
Mar 10 09:08:42 	kernel 		igb1: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		igb1: Using 4 RX queues 4 TX queues
Mar 10 09:08:42 	kernel 		igb1: Using MSI-X interrupts with 5 vectors
Mar 10 09:08:42 	kernel 		igb1: Ethernet address: 00:1b:21:27:12:37
Mar 10 09:08:42 	kernel 		igb1: netmap queues/slots: TX 4/1024, RX 4/1024
Mar 10 09:08:42 	kernel 		isab0: <PCI-ISA bridge> at device 31.0 on pci0
Mar 10 09:08:42 	kernel 		isa0: <ISA bus> on isab0
Mar 10 09:08:42 	kernel 		pci0: <memory> at device 31.2 (no driver attached)
Mar 10 09:08:42 	kernel 		hdac0: <Intel Sunrise Point HDA Controller> mem 0xe1344000-0xe1347fff,0xe1330000-0xe133ffff at device 31.3 on pci0
Mar 10 09:08:42 	kernel 		em0: <Intel(R) I219-LM SPT-H(2)> mem 0xe1300000-0xe131ffff at device 31.6 on pci0
Mar 10 09:08:42 	kernel 		em0: EEPROM V0.8-4
Mar 10 09:08:42 	kernel 		em0: Using 1024 TX descriptors and 1024 RX descriptors
Mar 10 09:08:42 	kernel 		em0: Using an MSI interrupt
Mar 10 09:08:42 	kernel 		em0: Ethernet address: 70:5a:0f:3b:19:12
Mar 10 09:08:42 	kernel 		em0: netmap queues/slots: TX 1/1024, RX 1/1024
Mar 10 09:08:42 	kernel 		acpi_button0: <Sleep Button> on acpi0
Mar 10 09:08:42 	kernel 		acpi_button1: <Power Button> on acpi0
Mar 10 09:08:42 	kernel 		atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Mar 10 09:08:42 	kernel 		atkbd0: <AT Keyboard> irq 1 on atkbdc0
Mar 10 09:08:42 	kernel 		kbd0 at atkbd0
Mar 10 09:08:42 	kernel 		atkbd0: [GIANT-LOCKED]
Mar 10 09:08:42 	kernel 		acpi_syscontainer0: <System Container> on acpi0
Mar 10 09:08:42 	kernel 		uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Mar 10 09:08:42 	kernel 		hwpstate_intel0: <Intel Speed Shift> on cpu0
Mar 10 09:08:42 	kernel 		hwpstate_intel1: <Intel Speed Shift> on cpu1
Mar 10 09:08:42 	kernel 		hwpstate_intel2: <Intel Speed Shift> on cpu2
Mar 10 09:08:42 	kernel 		hwpstate_intel3: <Intel Speed Shift> on cpu3
Mar 10 09:08:42 	kernel 		Timecounter "TSC-low" frequency 1595996890 Hz quality 1000
Mar 10 09:08:42 	kernel 		Timecounters tick every 1.000 msec
Mar 10 09:08:42 	kernel 		ZFS filesystem version: 5
Mar 10 09:08:42 	kernel 		ZFS storage pool version: features support (5000)
Mar 10 09:08:42 	kernel 		hdacc0: <Realtek ALC221 HDA CODEC> at cad 0 on hdac0
Mar 10 09:08:42 	kernel 		hdaa0: <Realtek ALC221 Audio Function Group> at nid 1 on hdacc0
Mar 10 09:08:42 	kernel 		pcm0: <Realtek ALC221 (Analog)> at nid 23 and 26,27 on hdaa0
Mar 10 09:08:42 	kernel 		pcm1: <Realtek ALC221 (Analog 2.0+HP)> at nid 20,33 on hdaa0
Mar 10 09:08:42 	kernel 		hdacc1: <Intel Skylake HDA CODEC> at cad 2 on hdac0
Mar 10 09:08:42 	kernel 		hdaa1: <Intel Skylake Audio Function Group> at nid 1 on hdacc1
Mar 10 09:08:42 	kernel 		pcm2: <Intel Skylake (HDMI/DP 8ch)> at nid 3 on hdaa1
Mar 10 09:08:42 	kernel 		Trying to mount root from zfs:pfSense/ROOT/default []...
Mar 10 09:08:42 	kernel 		ugen0.1: <Intel XHCI root HUB> at usbus0
Mar 10 09:08:42 	kernel 		uhub0 on usbus0
Mar 10 09:08:42 	kernel 		uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Mar 10 09:08:42 	kernel 		Root mount waiting for: usbus0 CAM
Mar 10 09:08:42 	kernel 		uhub0: 24 ports with 24 removable, self powered
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		Root mount waiting for: CAM
Mar 10 09:08:42 	kernel 		ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
Mar 10 09:08:42 	kernel 		ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Mar 10 09:08:42 	kernel 		ses0: SEMB SES Device
Mar 10 09:08:42 	kernel 		ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Mar 10 09:08:42 	kernel 		ada0: <KingFast HP2818E3> ACS-2 ATA SATA 3.x device
Mar 10 09:08:42 	kernel 		ada0: Serial Number CI0311M000805
Mar 10 09:08:42 	kernel 		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 4096bytes)
Mar 10 09:08:42 	kernel 		ada0: Command Queueing enabled
Mar 10 09:08:42 	kernel 		ada0: 244198MB (500118192 512 byte sectors)
Mar 10 09:08:42 	kernel 		ses0: pass0,ada0 in 'Slot 00', SATA Slot: scbus0 target 0
Mar 10 09:08:42 	kernel 		ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Mar 10 09:08:42 	kernel 		ada1: <SATA SSD SBFM61.5> ACS-4 ATA SATA 3.x device
Mar 10 09:08:42 	kernel 		ada1: Serial Number 23052925600661
Mar 10 09:08:42 	kernel 		ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Mar 10 09:08:42 	kernel 		ada1: Command Queueing enabled
Mar 10 09:08:42 	kernel 		ada1: 244198MB (500118192 512 byte sectors)
Mar 10 09:08:42 	kernel 		ses0: pass1,ada1 in 'Slot 01', SATA Slot: scbus1 target 0
Mar 10 09:08:42 	kernel 		CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
Mar 10 09:08:42 	kernel 		Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Mar 10 09:08:42 	kernel 		Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Mar 10 09:08:42 	kernel 		Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Mar 10 09:08:42 	kernel 		AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Mar 10 09:08:42 	kernel 		AMD Features2=0x121<LAHF,ABM,Prefetch>
Mar 10 09:08:42 	kernel 		Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Mar 10 09:08:42 	kernel 		Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Mar 10 09:08:42 	kernel 		XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Mar 10 09:08:42 	kernel 		IA32_ARCH_CAPS=0xc04<RSBA>
Mar 10 09:08:42 	kernel 		VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Mar 10 09:08:42 	kernel 		TSC: P-state invariant, performance statistics
Mar 10 09:08:42 	kernel 		pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xe134b000-0xe134bfff at device 20.2 on pci0
Mar 10 09:08:42 	kernel 		ichsmb0: <Intel Sunrise Point-H SMBus controller> port 0xefa0-0xefbf mem 0xe134e000-0xe134e0ff at device 31.4 on pci0
Mar 10 09:08:42 	kernel 		smbus0: <System Management Bus> on ichsmb0
Mar 10 09:08:42 	kernel 		lo0: link state changed to UP
Mar 10 09:08:45 	check_reload_status 	473 	Linkup starting igb1
Mar 10 09:08:45 	kernel 		igb1: link state changed to UP
Mar 10 09:08:51 	check_reload_status 	473 	rc.newwanip starting igb1
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: rc.newwanip: Info: starting on igb1.
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: rc.newwanip: on (IP address: 100.66.98.149) (interface: WAN[wan]) (real interface: igb1).
Mar 10 09:08:52 	php-fpm 	409 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Mar 10 09:08:53 	kernel 		[fib_algo] inet.0 (bsearch4#20) rebuild_fd_flm: switching algo to radix4_lockless
Mar 10 09:08:53 	php-cgi 	508 	rc.bootup: Resyncing OpenVPN instances.
Mar 10 09:08:54 	kernel 		pflog0: promiscuous mode enabled
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: Removing static route for monitor 8.8.8.8 and adding a new route through 100.66.96.1
Mar 10 09:08:54 	kernel 		.done.
Mar 10 09:08:54 	kernel 		done.
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: Gateway, NONE AVAILABLE
Mar 10 09:08:54 	php-cgi 	508 	rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	php-cgi 	508 	rc.bootup: sync unbound done.
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	kernel 		done.
Mar 10 09:08:55 	check_reload_status 	473 	Linkup starting em0
Mar 10 09:08:55 	kernel 		em0: link state changed to UP
Mar 10 09:08:56 	php-fpm 	409 	/rc.linkup: Ignoring link event during boot sequence.
Mar 10 09:09:02 	php-cgi 	508 	rc.bootup: NTPD is starting up.
Mar 10 09:09:02 	kernel 		done.
Mar 10 09:09:03 	kernel 		done.
Mar 10 09:09:03 	kernel 		done.
Mar 10 09:09:03 	check_reload_status 	473 	Updating all dyndns
Mar 10 09:09:03 	php-cgi 	508 	rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).'
Mar 10 09:09:03 	kernel 		.done.
Mar 10 09:09:08 	php-cgi 	508 	rc.bootup: Creating rrd update script
Mar 10 09:09:08 	kernel 		done.
Mar 10 09:09:08 	syslogd 		exiting on signal 15
Mar 10 09:09:08 	syslogd 		kernel boot file is /boot/kernel/kernel
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: Restarting/Starting all packages.
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: Starting service avahi
Mar 10 09:09:09 	php-fpm 	409 	/rc.start_packages: [zeek] Starting service...
Mar 10 09:09:09 	SnortStartup 	85723 	Snort START for WAN(igb1)...
Mar 10 09:09:09 	avahi-daemon 	82896 	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Mar 10 09:09:09 	kernel 		igb1: promiscuous mode enabled
Mar 10 09:09:09 	avahi-daemon 	82896 	Successfully dropped root privileges.
Mar 10 09:09:09 	avahi-daemon 	82896 	avahi-daemon 0.8 starting up.
Mar 10 09:09:09 	avahi-daemon 	82896 	No service file found in /usr/local/etc/avahi/services.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface lo0.IPv4 with address 127.0.0.1.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface lo0.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface em0.IPv4 with address 192.168.1.1.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface em0.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Joining mDNS multicast group on interface igb1.IPv4 with address 100.66.98.149.
Mar 10 09:09:09 	avahi-daemon 	82896 	New relevant interface igb1.IPv4 for mDNS.
Mar 10 09:09:09 	avahi-daemon 	82896 	Network interface enumeration completed.
Mar 10 09:09:09 	avahi-daemon 	82896 	Server startup complete. Host name is pfSense.local. Local service cookie is 804603802.
Mar 10 09:09:12 	php-fpm 	409 	/rc.start_packages: [zeek] Adding cronjobs ...
Mar 10 09:09:14 	root 	57312 	Bootup complete
Mar 10 09:09:15 	login 	58466 	login on ttyv0 as root
Mar 10 09:09:15 	sshguard 	60700 	Now monitoring attacks.
Mar 10 09:09:17 	arpwatch 	87523 	listening on igb1
Mar 10 09:09:34 	php-fpm 	410 	/diag_packet_capture.php: Successful login for user 'admin' from: 192.168.1.122 (Local Database)

And the network appears to be funtioning correctly now.

Sergei_Shablovsky

Sorry to insert my 5c on this, but:

BUY used/new Intel-based NIC FROM WELL-KNOWN DEVELOPER (Intel, IBM preferable) on eBay/local store.

on pfSense

RESET BIOS/UEFI to default
UPGRADE both(!, upper and lower) IMAGE OF BIOS/UEFI from official MB’s manufacturer web;
PULL all PSU OUT (if they are hot swap), internal CR2032 battery, wait 1min, take it all in place back;
DISABLE ALL POWER MANAGEMENT ON pfSense’s MOTHERBOARD (especially for CPU, PCI and NICs)!
COLD Restart.

REPLACE all NICs that installed in main pfSense.
Starting Kali Linux (or Win10/11) from USB-drive, and:

check that ALL NICs able to receive IPs from uplink ISP;
check line rate/error free on each NIC by iperf3 public servers;

As a result on this point You are known that NICs and cables are on good working order.

INSTALL FRESH pfSense with option “Use the previous configuration file”.

RE-ASSIGN INTERFACES to new in local-attached keyb/monitor, COM-port terminal, or WebGUI.

COLD RESTART.

And see if issue still exist.

P.S.
You spend so much time on searching on this forum, replying, googling the same issue, so MUCH FASTER would be buying the few new NICs (ok, but it used;) check hardware first.
Anyway, You not be disappointed by this NICs upgrade in a future.

bmeeks

@Wylbur said in Back to odd problem -- lose WAN at random points with a week or more between events:

Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 38 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 f8 17 e9 40 08 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 28 e0 19 ab 40 0a 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2a 28 40 00 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2c cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 10 2e cf 40 1d 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 30 40 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 18 70 fc 03 40 09 00 00 00 00 00
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): CAM status: Auto-Sense Retrieval Failed
Mar 10 01:55:17 kernel (ada1:ahcich1:0:0:0): Error 5, Unretryable error
Mar 10 09:08:42 syslogd kernel boot file is /boot/kernel/kernel
Mar 10 09:08:42 kernel ---<<BOOT>>---

These errors indicate a failing disk drive (whether it's an SSD or an old spinning surface, it is failing).

You need to be sure you have a backup of the firewall configuration on separate media (such as a USB stick), then replace the failing drive and reinstall pfSense from an install image restoring you config during the install process.

keyser

@Wylbur I think you need to have a look in the DHCP log and see if the issue arises when DHCLIENT (WAN DHCP client) tries to renew the DHCP lease. Some ISP’s are quite picky with other hardware on their infrastructure, and require a quite strict DHCP client configuration.
You know that your DISCOVER/OFFER/REQUEST/ACK (new DHCP lease works), but does a renewal of an existing lease?