Solid Stable [2.4.5-RELEASE] - Dropping LAN, Requires Reboot
-
My pfSense has been up an running over a year. I keep it updated, backed up, etc. After making no changes, last week I lost internet. Traced it down to the basement, and rebooted a switch. Ping to the firewall LAN gateway came up, but still no internet. Rebooted the firewall and we were back in business.
This happened 5 or 6 more times over 3 days. I thought it might be the switch. I ordered and installed a brand new one. After 2 days, the problem happened again. When I lose network, I reboot the switch, and I can ping LAN devices but no internet. I then reboot the firewall, and internet comes back up.
Prior to rebooting the firewall I cannot access the GUI or ping any IP on the device.
What can I do to troubleshoot this?
This is the logged events from when I rebooted the switch:
Jan 22 13:42:33 php-fpm 59678 /rc.linkup: Hotplug event detected for VLAN101(opt3) static IP (192.168.101.1 ) Jan 22 13:42:32 check_reload_status Linkup starting igb1.101 Jan 22 13:42:32 check_reload_status Linkup starting igb1 Jan 22 13:42:32 kernel igb1.101: link state changed to UP Jan 22 13:42:32 kernel igb1: link state changed to UP Jan 22 13:42:27 check_reload_status Reloading filter Jan 22 13:42:27 check_reload_status Reloading filter Jan 22 13:42:27 avahi-daemon 13670 Interface igb1.IPv4 no longer relevant for mDNS. Jan 22 13:42:27 avahi-daemon 13670 Leaving mDNS multicast group on interface igb1.IPv4 with address 10.10.10.1. Jan 22 13:42:27 avahi-daemon 13670 Joining mDNS multicast group on interface igb1.IPv4 with address 10.10.10.1. Jan 22 13:42:27 avahi-daemon 13670 Leaving mDNS multicast group on interface igb1.IPv4 with address 192.168.2.1. Jan 22 13:42:27 php-fpm 352 /rc.linkup: Hotplug event detected for VLAN101(opt3) static IP (192.168.101.1 ) Jan 22 13:42:27 php-fpm 59678 /rc.linkup: DEVD Ethernet detached event for lan Jan 22 13:42:26 check_reload_status Linkup starting igb1.101 Jan 22 13:42:26 kernel igb1.101: link state changed to DOWN Jan 22 13:42:26 kernel igb1: link state changed to DOWN Jan 22 13:42:26 check_reload_status Linkup starting igb1 Jan 22 12:30:45 php-cgi rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP_v4 does not need updating. Jan 22 12:30:45 php-cgi rc.update_urltables: /etc/rc.update_urltables: pfB_Whitelist_v4 does not need updating. Jan 22 12:30:45 php-cgi rc.update_urltables: /etc/rc.update_urltables: pfB_Level4_v4 does not need updating.And here is the log from when I reboot the pfSense:
Jan 22 13:57:59 kernel Origin="GenuineIntel" Id=0x40651 Family=0x6 Model=0x45 Stepping=1 Jan 22 13:57:59 kernel CPU: Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz (1895.65-MHz K8-class CPU) Jan 22 13:57:59 kernel VT(vga): resolution 640x480 Jan 22 13:57:59 kernel FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1) Jan 22 13:57:59 kernel root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense amd64 Jan 22 13:57:59 kernel FreeBSD 11.3-STABLE #243 abf8cba50ce(RELENG_2_4_5): Tue Jun 2 17:53:37 EDT 2020 Jan 22 13:57:59 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Jan 22 13:57:59 kernel The Regents of the University of California. All rights reserved. Jan 22 13:57:59 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jan 22 13:57:59 kernel Copyright (c) 1992-2020 The FreeBSD Project. Jan 22 13:57:59 syslogd kernel boot file is /boot/kernel/kernel Jan 22 13:57:01 syslogd exiting on signal 15 Jan 22 13:43:16 check_reload_status Reloading filter Jan 22 13:43:16 php-fpm 351 /rc.newwanipv6: Removing static route for monitor fe80::201:5cff:fead:ec46 and adding a new route through fe80::201:5cff:fead:ec46%igb0 Jan 22 13:42:41 php-fpm 351 /rc.newwanipv6: rc.newwanipv6: on (IP address: 2001:XXX:d81:3528:645c) (interface: wan) (real interface: igb0). Jan 22 13:42:41 php-fpm 351 /rc.newwanipv6: rc.newwanipv6: Info: starting on igb0. Jan 22 13:42:40 check_reload_status Reloading filter Jan 22 13:42:40 check_reload_status updating dyndns lan Jan 22 13:42:40 rtsold Received RA specifying route fe80::201:5cff:fead:ec46 for interface wan(igb0) Jan 22 13:42:34 php-fpm 59678 /rc.newwanip: rc.newwanip: on (IP address: 192.168.101.1) (interface: VLAN101[opt3]) (real interface: igb1.101). Jan 22 13:42:34 php-fpm 59678 /rc.newwanip: rc.newwanip: Info: starting on igb1.101. Jan 22 13:42:33 check_reload_status Restarting ipsec tunnels Jan 22 13:42:33 check_reload_status Reloading filter Jan 22 13:42:33 check_reload_status rc.newwanip starting igb1.101 -
I also see this crazy gap in the log, though the internet was up throughout this gap.
Jan 22 12:30:45 php-cgi rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating. Jan 22 12:30:45 php-cgi rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates Jan 22 12:30:00 php-cgi rc.update_urltables: /etc/rc.update_urltables: Sleeping for 45 seconds. Jan 22 12:30:00 php-cgi rc.update_urltables: /etc/rc.update_urltables: Starting up. Jan 22 04:00:19 php /usr/local/sbin/execacb.php: End of configuration backup to https://acb.netgate.com/save (success). Jan 22 04:00:00 php /usr/local/sbin/execacb.php: Beginning configuration backup to https://acb.netgate.com/save Jan 22 04:00:00 check_reload_status Syncing firewall Jan 22 01:01:05 php-cgi rc.dyndns.update: phpDynDNS (plex.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jan 22 01:01:04 php-cgi rc.dyndns.update: phpDynDNS (terraria.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jan 22 01:01:02 php-cgi rc.dyndns.update: phpDynDNS (pfsense.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. -
Also found these around the same time.
Jan 22 13:58:14 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.10.1 bind_addr 192.168.10.1 identifier "OVPN_VPNV4 " Jan 22 13:58:14 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr fe80::201:XXX:ec46%igb0 bind_addr fe80::XXX:78f%igb0 identifier "WAN1_DHCP6 " Jan 22 13:58:14 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.XXX.XXX.1 bind_addr 73.XXX.XXX.185 identifier "WAN1_DHCP " Jan 22 13:58:08 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.10.1 bind_addr 192.168.10.1 identifier "OVPN_VPNV4 " Jan 22 13:58:08 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.XXX.XXX.1 bind_addr 73.XXX.XXX.185 identifier "WAN1_DHCP " Jan 22 13:58:07 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.10.1 bind_addr 192.168.10.1 identifier "OVPN_VPNV4 " Jan 22 13:58:07 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.XXX.XXX.1 bind_addr 73.XXX.XXX.185 identifier "WAN1_DHCP " Jan 22 13:58:05 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.10.1 bind_addr 192.168.10.1 identifier "OVPN_VPNV4 " Jan 22 13:58:05 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.XXX.XXX.1 bind_addr 73.XXX.XXX.185 identifier "WAN1_DHCP " Jan 22 13:43:16 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.10.1 bind_addr 192.168.10.1 identifier "OVPN_VPNV4 " Jan 22 13:43:16 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr fe80::201:XXX:ec46%igb0 bind_addr fe80::XXX:78f%igb0 identifier "WAN1_DHCP6 " Jan 22 13:43:16 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 73.XXX.XXX.1 bind_addr 73.XXX.XXX.185 identifier "WAN1_DHCP " -
If nothing is happening then nothing gets logged. A low rate system log is often a sign of good stability not a problem.
Those last logs are all dpinger starting which happens if an interface goes down and then comes back up or you apply a change to it etc.
I see in there that you have an OpenVPN gateway. Do you have the WAN gateway set as default in System > Routing > Gateways? If not it might have switched to a different default route when your WAN went down resulting in no connectivity.
I don't see that logged specifically but those logs are incomplete.
Steve
