pfSense loses WAN/LAN connectivity - Need Help Checking Logs
-
I have a pfsense 2.5.2 install on an HP desktop CPU i3-4160 @ 3.6Ghz
8GB Ram and it appears it just crashed my internet after 7 days. I am coming from 2 years of internet crashing every 3-4 days with using Orbi as a router. 3 different ISPs and upgraded from an Orbi RBR50 (new 7 years ago) to an Orbi RBR850 (in November 2021) all being router mode. I finally made the switch and put my new Orbi RBR850 to AP mode and 7 days later the pfsense router caused it to disconnect again. I rebooted the Orbi, ATT fiber modem and it did not connect. Finally rebooted the pfsense and it came back online. Any help appreciated! Thanks y’all.pfSense v2.5.2
CPU i3-4160 @ 3.6Ghz
8GB Ram -
@rmoran
Here's some of my Log-System-Last 1000 entries file. This is when my son noticed the outage last night.Feb 20 17:55:11 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:55:15 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 17:55:23 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:55:28 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 17:55:31 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:55:35 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 17:56:00 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:56:07 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 17:56:29 pfSense kernel: arp: 192.168.1.250 moved from 6c:cd:d6:d6:bd:56 to 9c:8e:cd:28:66:fe on re0 Feb 20 17:56:35 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:56:37 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 17:56:39 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 17:56:43 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 18:36:54 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 20 18:37:34 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 20 20:53:32 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:53:32 pfSense kernel: re0: link state changed to DOWN Feb 20 20:53:33 pfSense php-fpm[76369]: /rc.linkup: DEVD Ethernet detached event for lan Feb 20 20:53:33 pfSense check_reload_status[379]: Reloading filter Feb 20 20:53:44 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:53:44 pfSense kernel: re0: link state changed to UP Feb 20 20:53:45 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet attached event for lan Feb 20 20:53:45 pfSense php-fpm[349]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 20 20:53:45 pfSense php-fpm[349]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 20:53:45 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 20:53:50 pfSense check_reload_status[379]: updating dyndns lan Feb 20 20:53:50 pfSense check_reload_status[379]: Reloading filter Feb 20 20:53:58 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:53:58 pfSense kernel: re0: link state changed to DOWN Feb 20 20:53:59 pfSense php-fpm[76369]: /rc.linkup: DEVD Ethernet detached event for lan Feb 20 20:53:59 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:00 pfSense kernel: re0: link state changed to UP Feb 20 20:54:00 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:01 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet attached event for lan Feb 20 20:54:01 pfSense php-fpm[349]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 20 20:54:01 pfSense php-fpm[349]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 20:54:01 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 20:54:05 pfSense check_reload_status[379]: updating dyndns lan Feb 20 20:54:05 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:11 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:11 pfSense kernel: re0: link state changed to DOWN Feb 20 20:54:12 pfSense php-fpm[76369]: /rc.linkup: DEVD Ethernet detached event for lan Feb 20 20:54:12 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:14 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:14 pfSense kernel: re0: link state changed to UP Feb 20 20:54:15 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet attached event for lan Feb 20 20:54:15 pfSense php-fpm[349]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 20 20:54:15 pfSense php-fpm[349]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 20:54:15 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 20:54:19 pfSense check_reload_status[379]: updating dyndns lan Feb 20 20:54:19 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:23 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:23 pfSense kernel: re0: link state changed to DOWN Feb 20 20:54:24 pfSense php-fpm[76369]: /rc.linkup: DEVD Ethernet detached event for lan Feb 20 20:54:24 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:26 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:26 pfSense kernel: re0: link state changed to UP Feb 20 20:54:27 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet attached event for lan Feb 20 20:54:27 pfSense php-fpm[349]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 20 20:54:27 pfSense php-fpm[349]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 20:54:27 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 20:54:29 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:29 pfSense kernel: re0: link state changed to DOWN Feb 20 20:54:31 pfSense check_reload_status[379]: updating dyndns lan Feb 20 20:54:31 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:31 pfSense php-fpm[350]: /rc.linkup: DEVD Ethernet detached event for lan Feb 20 20:54:31 pfSense check_reload_status[379]: Reloading filter Feb 20 20:54:32 pfSense kernel: re0: link state changed to UP Feb 20 20:54:32 pfSense check_reload_status[379]: Linkup starting re0 Feb 20 20:54:33 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet attached event for lan Feb 20 20:54:33 pfSense php-fpm[349]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 20 20:54:33 pfSense php-fpm[349]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 20:54:33 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 20:54:38 pfSense check_reload_status[379]: updating dyndns lan Feb 20 20:54:38 pfSense check_reload_status[379]: Reloading filter Feb 20 21:03:58 pfSense kernel: em0: link state changed to DOWN Feb 20 21:03:58 pfSense check_reload_status[379]: Linkup starting em0 Feb 20 21:03:59 pfSense php-fpm[350]: /rc.linkup: DEVD Ethernet detached event for wan Feb 20 21:04:01 pfSense php-fpm[350]: /rc.linkup: Shutting down Router Advertisment daemon cleanly Feb 20 21:04:01 pfSense check_reload_status[379]: Reloading filter Feb 20 21:04:13 pfSense rc.gateway_alarm[7328]: >>> Gateway alarm: WAN_DHCP (Addr:X.X.X.X Alarm:1 RTT:1.525ms RTTsd:.207ms Loss:22%) Feb 20 21:04:13 pfSense check_reload_status[379]: updating dyndns WAN_DHCP Feb 20 21:04:13 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 21:04:13 pfSense check_reload_status[379]: Restarting OpenVPN tunnels/interfaces Feb 20 21:04:13 pfSense check_reload_status[379]: Reloading filter Feb 20 21:04:59 pfSense check_reload_status[379]: Linkup starting em0 Feb 20 21:04:59 pfSense kernel: em0: link state changed to UP Feb 20 21:05:00 pfSense php-fpm[40167]: /rc.linkup: DEVD Ethernet attached event for wan Feb 20 21:05:00 pfSense php-fpm[40167]: /rc.linkup: HOTPLUG: Configuring interface wan Feb 20 21:05:05 pfSense check_reload_status[379]: rc.newwanip starting em0 Feb 20 21:05:05 pfSense php-fpm[40167]: /rc.linkup: calling interface_dhcpv6_configure. Feb 20 21:05:05 pfSense php-fpm[40167]: /rc.linkup: Accept router advertisements on interface em0 Feb 20 21:05:05 pfSense php-fpm[40167]: /rc.linkup: Starting rtsold process Feb 20 21:05:06 pfSense php-fpm[350]: /rc.newwanip: rc.newwanip: Info: starting on em0. Feb 20 21:05:06 pfSense php-fpm[350]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.2.64) (interface: WAN[wan]) (real interface: em0). Feb 20 21:05:06 pfSense php-fpm[350]: /rc.newwanip: Accept router advertisements on interface em0 Feb 20 21:05:06 pfSense php-fpm[350]: /rc.newwanip: Starting rtsold process Feb 20 21:05:07 pfSense php-fpm[40167]: /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Feb 20 21:05:07 pfSense php-fpm[40167]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 21:05:07 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 20 21:05:08 pfSense php-fpm[350]: /rc.newwanip: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_em0.pid -M /var/etc/rtsold_em0_script.sh -O /var/etc/rtsold_em0_script.sh em0' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Feb 20 21:05:09 pfSense php-fpm[350]: /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 20 21:05:09 pfSense php-fpm[350]: /rc.newwanip: IP Address has changed, killing states on former IP Address X.X.X.X. Feb 20 21:05:10 pfSense php-fpm[350]: /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1645412710] unbound[48466:0] error: bind: address already in use [1645412710] unbound[48466:0] fatal error: could not open ports'
This morning we lost wifi connection for a while with the AP. This is was the time frame when it dropped the connection. -----------
Feb 21 05:49:56 pfSense check_reload_status[379]: Linkup starting re0 Feb 21 05:49:56 pfSense kernel: re0: watchdog timeout Feb 21 05:49:56 pfSense kernel: re0: link state changed to DOWN Feb 21 05:49:57 pfSense php-fpm[349]: /rc.linkup: DEVD Ethernet detached event for lan Feb 21 05:49:57 pfSense check_reload_status[379]: Reloading filter Feb 21 05:50:00 pfSense check_reload_status[379]: Linkup starting re0 Feb 21 05:50:00 pfSense kernel: re0: link state changed to UP Feb 21 05:50:01 pfSense php-fpm[350]: /rc.linkup: DEVD Ethernet attached event for lan Feb 21 05:50:01 pfSense php-fpm[350]: /rc.linkup: HOTPLUG: Configuring interface lan Feb 21 05:50:01 pfSense php-fpm[350]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Feb 21 05:50:01 pfSense check_reload_status[379]: Restarting ipsec tunnels Feb 21 05:50:06 pfSense check_reload_status[379]: updating dyndns lan Feb 21 05:50:06 pfSense check_reload_status[379]: Reloading filter Feb 21 05:50:10 pfSense kernel: arp: 192.168.1.24 moved from 9c:8e:cd:28:50:d3 to 9c:8e:cd:28:66:fe on re0 Feb 21 05:51:23 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 21 05:51:30 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:51:35 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 21 05:51:39 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:51:43 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:51:45 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:51:48 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:51:51 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 21 05:52:02 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:52:10 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 21 05:52:40 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:52:43 pfSense kernel: arp: 192.168.1.4 moved from 80:cc:9c:12:45:a7 to 82:cc:9c:12:45:aa on re0 Feb 21 05:52:45 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:52:47 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:52:49 pfSense kernel: arp: 192.168.1.4 moved from 82:cc:9c:12:45:aa to 80:cc:9c:12:45:a7 on re0 Feb 21 05:55:29 pfSense php-fpm[349]: /index.php: Successful login for user 'admin' from: 192.168.1.219 (Local Database) Feb 21 06:16:13 pfSense php-fpm[350]: /system_update_settings.php: Successful login for user 'admin' from: 192.168.1.21 (Local Database) Feb 21 06:29:57 pfSense check_reload_status[379]: Syncing firewall Feb 21 06:29:57 pfSense syslogd: exiting on signal 15 Feb 21 06:29:57 pfSense syslogd: kernel boot file is /boot/kernel/kernel Feb 21 06:29:58 pfSense nginx: 2022/02/21 06:29:58 [error] 62812#100561: send() failed (54: Connection reset by peer) Feb 21 06:30:17 pfSense check_reload_status[379]: Syncing firewall Feb 21 06:30:17 pfSense syslogd: exiting on signal 15 Feb 21 06:30:17 pfSense syslogd: kernel boot file is /boot/kernel/kernel Feb 21 06:30:27 pfSense check_reload_status[379]: Syncing firewall Feb 21 06:30:27 pfSense syslogd: exiting on signal 15 Feb 21 06:30:27 pfSense syslogd: kernel boot file is /boot/kernel/kernel Feb 21 06:30:27 pfSense nginx: 2022/02/21 06:30:27 [error] 62812#100561: send() failed (54: Connection reset by peer) Feb 21 06:31:37 pfSense check_reload_status[379]: Syncing firewall Feb 21 06:31:37 pfSense syslogd: exiting on signal 15 Feb 21 06:31:37 pfSense syslogd: kernel boot file is /boot/kernel/kernel Feb 21 06:41:21 pfSense check_reload_status[379]: Linkup starting re1 Feb 21 06:41:21 pfSense kernel: re1: link state changed to DOWN
-
How are those interfaces physically connected?
You have log entries there showing the NICs losing link, like the cable was disconnected or whatever they are attached to rebooted.
Now I would normally call into question the Realtek NICs have but there are also logs for em0 losing link.
Steve