Why did my WAN drop and not reconnect?
-
Hi,
Last night my WAN dropped out at 04.09am and didn't come back by itself. I had to reboot the router when I woke up at 7.44. The router itself was still fully responsive and accessible from the LAN. It is 2.7 CE running on bare metal with Intel i226 Ethernet ports.
I'll post the logs further down, with a bit before and after.
I also contacted my ISP, who advised :
Hi Chris
Our connection logs show...
2023-11-02T04:09:54.152615+00:00 clueless radius-acct: BBEUXXXXXXXX
Stopped XXXXXXX LostCarrier
2023-11-02T04:09:54.222180+00:00 clueless bgpfeed: XXXXXXX Route
2001:XXX:XXX:XXXX::/64 now downOther fibre circuits in XXX do not show the same outage, there's no
planned work by us or BT and no incident reports.Bit of a mystery, sorry not much help.
Here are the PFsense PPP logs. Below them are my general system logs too.
Oct 30 21:10:58 ppp 11723 [wan] IFACE: Add description "WAN"
Oct 30 21:10:58 ppp 11723 [wan] IPV6CP: rec'd Configure Request #0 (Ack-Rcvd)
Oct 30 21:10:58 ppp 11723 [wan] IPV6CP: SendConfigAck #0
Oct 30 21:10:58 ppp 11723 [wan] IPV6CP: state change Ack-Rcvd --> Opened
Oct 30 21:10:58 ppp 11723 [wan] IPV6CP: LayerUp
Oct 30 21:10:58 ppp 11723 [wan] 7e2b:e1ff:fe13:eabb -> 0203:97ff:feba:0900
Nov 2 04:09:24 ppp 11723 caught fatal signal TERM
Nov 2 04:09:24 ppp 11723 [wan] IFACE: Close event
Nov 2 04:09:24 ppp 11723 [wan] IPCP: Close event
Nov 2 04:09:24 ppp 11723 [wan] IPCP: state change Opened --> Closing
Nov 2 04:09:24 ppp 11723 [wan] IPCP: SendTerminateReq #4
Nov 2 04:09:24 ppp 11723 [wan] error writing len 8 frame to b0: Network is down
Nov 2 04:09:24 ppp 11723 [wan] IPCP: LayerDown
Nov 2 04:09:27 ppp 16129 Multi-link PPP daemon for FreeBSD
Nov 2 04:09:27 ppp 16129 process 16129 started, version 5.9
Nov 2 04:09:27 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:28 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:29 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:52 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:53 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:54 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:55 ppp 11723 [wan] IFACE: Removing IPv4 address from pppoe1 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: Close event
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: state change Opened --> Closing
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: SendTerminateReq #2
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: LayerDown
Nov 2 04:09:55 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Down event
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Rename interface pppoe1 to pppoe1
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Set description "WAN"
Nov 2 04:09:56 ppp 11723 [wan] IPCP: SendTerminateReq #5
Nov 2 04:09:56 ppp 11723 [wan_link0] PPPoE: connection closed
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: DOWN event
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: giving up after 0 reconnection attempts
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: Close event
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: state change Opened --> Closing
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: Leave bundle "wan"
Nov 2 04:09:56 ppp 11723 [wan] Bundle: Status update: up 0 links, total bandwidth 9600 bps
Nov 2 04:09:56 ppp 11723 [wan] IPCP: Close event
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: Close event
Nov 2 04:09:56 ppp 11723 [wan] IPCP: Down event
Nov 2 04:09:56 ppp 11723 [wan] IPCP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan] IPCP: state change Closing --> Initial
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: Down event
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan] Bundle: No NCPs left. Closing links...
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: state change Closing --> Initial
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: SendTerminateReq #3
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: LayerDown
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: Down event
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: state change Closing --> Initial
Nov 2 04:09:56 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:57 ppp 16129 can't lock /var/run/pppoe_wan.pid after 30 attempts
Nov 2 04:09:58 ppp 11723 [wan] Bundle: Shutdown
Nov 2 04:09:58 ppp 11723 [wan_link0] Link: Shutdown
Nov 2 04:09:58 ppp 11723 process 11723 terminated
Nov 2 07:44:11 ppp 11724 Multi-link PPP daemon for FreeBSD
Nov 2 07:44:11 ppp 11724 process 11724 started, version 5.9
Nov 2 07:44:11 ppp 11724 web: web is not running
Nov 2 07:44:11 ppp 11724 [wan] Bundle: Interface ng0 created
Nov 2 07:44:11 ppp 11724 [wan_link0] Link: OPEN event
Nov 2 07:44:11 ppp 11724 [wan_link0] LCP: Open event
Nov 2 07:44:11 ppp 11724 [wan_link0] LCP: state change Initial --> Starting
Nov 2 07:44:11 ppp 11724 [wan_link0] LCP: LayerStart
Nov 2 07:44:11 ppp 11724 [wan_link0] PPPoE: Connecting to ''
Nov 2 07:44:18 ppp 11724 PPPoE: rec'd ACNAME "acc-aln7.li"
Nov 2 07:44:18 ppp 11724 [wan_link0] PPPoE: connection successful
Nov 2 07:44:18 ppp 11724 [wan_link0] Link: UP event
Nov 2 07:44:18 ppp 11724 [wan_link0] LCP: Up eventSystem logs
Nov 2 03:54:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 03:54:48 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 04:09:23 check_reload_status 417 Linkup starting igc0
Nov 2 04:09:23 kernel igc0: link state changed to DOWN
Nov 2 04:09:24 ppp 11723 caught fatal signal TERM
Nov 2 04:09:24 ppp 11723 [wan] IFACE: Close event
Nov 2 04:09:24 ppp 11723 [wan] IPCP: Close event
Nov 2 04:09:24 ppp 11723 [wan] IPCP: state change Opened --> Closing
Nov 2 04:09:24 ppp 11723 [wan] IPCP: SendTerminateReq #4
Nov 2 04:09:24 ppp 11723 [wan] error writing len 8 frame to b0: Network is down
Nov 2 04:09:24 ppp 11723 [wan] IPCP: LayerDown
Nov 2 04:09:24 php-cgi 9023 rc.kill_states: rc.kill_states: Removing states for IP xxx.xxx.xxx.xxx/32
Nov 2 04:09:26 check_reload_status 417 Linkup starting igc0
Nov 2 04:09:26 kernel igc0: link state changed to UP
Nov 2 04:09:27 check_reload_status 417 Reloading filter
Nov 2 04:09:27 ppp 16129 Multi-link PPP daemon for FreeBSD
Nov 2 04:09:27 ppp 16129 process 16129 started, version 5.9
Nov 2 04:09:27 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:27 php-fpm 76039 /rc.linkup: calling interface_dhcpv6_configure.
Nov 2 04:09:27 php-fpm 76039 /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_PPPOE'
Nov 2 04:09:27 php-fpm 76039 /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Nov 2 04:09:27 check_reload_status 417 Restarting IPsec tunnels
Nov 2 04:09:27 check_reload_status 417 updating dyndns wan
Nov 2 04:09:27 check_reload_status 417 Reloading filter
Nov 2 04:09:28 check_reload_status 417 Synching URL alias pfB_PRI1_v4
Nov 2 04:09:28 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:29 check_reload_status 417 Synching URL alias pfB_PRI1_v4
Nov 2 04:09:29 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:29 php-fpm 379 /rc.update_urltables: : Starting up.
Nov 2 04:09:29 php-fpm 379 /rc.update_urltables: : Sleeping for 40 seconds.
Nov 2 04:09:30 php-fpm 25002 /rc.update_urltables: : Starting up.
Nov 2 04:09:30 php-fpm 25002 /rc.update_urltables: : Sleeping for 48 seconds.
Nov 2 04:09:30 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:31 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:32 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:33 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:34 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:35 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:36 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:37 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:38 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:39 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:40 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:41 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:42 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:43 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:44 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:45 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:46 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:47 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:48 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:49 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:50 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:51 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:52 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:53 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:54 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:55 php-cgi 9023 rc.kill_states: rc.kill_states: Removing states for interface pppoe1
Nov 2 04:09:55 check_reload_status 417 Rewriting resolv.conf
Nov 2 04:09:55 ppp 11723 [wan] IFACE: Removing IPv4 address from pppoe1 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: Close event
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: state change Opened --> Closing
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: SendTerminateReq #2
Nov 2 04:09:55 ppp 11723 [wan] IPV6CP: LayerDown
Nov 2 04:09:55 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:56 php-cgi 66819 rc.kill_states: rc.kill_states: Removing states for IP fe80::7e2b:e1ff:fe13:eabb%pppoe1/32
Nov 2 04:09:56 php-cgi 66819 rc.kill_states: rc.kill_states: Removing states for interface pppoe1
Nov 2 04:09:56 check_reload_status 417 Rewriting resolv.conf
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Down event
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Rename interface pppoe1 to pppoe1
Nov 2 04:09:56 ppp 11723 [wan] IFACE: Set description "WAN"
Nov 2 04:09:56 ppp 11723 [wan] IPCP: SendTerminateReq #5
Nov 2 04:09:56 ppp 11723 [wan_link0] PPPoE: connection closed
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: DOWN event
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: giving up after 0 reconnection attempts
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: Close event
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: state change Opened --> Closing
Nov 2 04:09:56 ppp 11723 [wan_link0] Link: Leave bundle "wan"
Nov 2 04:09:56 ppp 11723 [wan] Bundle: Status update: up 0 links, total bandwidth 9600 bps
Nov 2 04:09:56 ppp 11723 [wan] IPCP: Close event
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: Close event
Nov 2 04:09:56 ppp 11723 [wan] IPCP: Down event
Nov 2 04:09:56 ppp 11723 [wan] IPCP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan] IPCP: state change Closing --> Initial
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: Down event
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan] Bundle: No NCPs left. Closing links...
Nov 2 04:09:56 ppp 11723 [wan] IPV6CP: state change Closing --> Initial
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: SendTerminateReq #3
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: LayerDown
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: Down event
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: LayerFinish
Nov 2 04:09:56 ppp 11723 [wan_link0] LCP: state change Closing --> Initial
Nov 2 04:09:56 ppp 16129 waiting for process 11723 to die...
Nov 2 04:09:57 ppp 16129 can't lock /var/run/pppoe_wan.pid after 30 attempts
Nov 2 04:09:58 ppp 11723 [wan] Bundle: Shutdown
Nov 2 04:09:58 ppp 11723 [wan_link0] Link: Shutdown
Nov 2 04:09:58 ppp 11723 process 11723 terminated
Nov 2 04:10:00 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 04:10:02 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 04:10:09 php-fpm 379 /rc.update_urltables: : Starting URL table alias updates
Nov 2 04:10:09 php-fpm 379 /rc.update_urltables: : ERROR: could not update pfB_PRI1_v4 content from https://127.0.0.1:443/pfblockerng/pfblockerng.php?pfb=pfB_PRI1_v4 <br />[ Abuse_Feodo_C2_v4, Abuse_IPBL_v4, Abuse_SSLBL_v4, CINS_army_v4, ET_Block_v4, ET_Comp_v4, ISC_Block_v4, Spamhaus_Drop_v4, Spamhaus_eDrop_v4, Talos_BL_v4 ]
Nov 2 04:10:18 php-fpm 25002 /rc.update_urltables: : Starting URL table alias updates
Nov 2 04:10:18 php-fpm 25002 /rc.update_urltables: : ERROR: could not update pfB_PRI1_v4 content from https://127.0.0.1:443/pfblockerng/pfblockerng.php?pfb=pfB_PRI1_v4 <br />[ Abuse_Feodo_C2_v4, Abuse_IPBL_v4, Abuse_SSLBL_v4, CINS_army_v4, ET_Block_v4, ET_Comp_v4, ISC_Block_v4, Spamhaus_Drop_v4, Spamhaus_eDrop_v4, Talos_BL_v4 ]
Nov 2 04:10:50 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 04:10:53 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 04:13:11 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 04:13:14 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 04:14:58 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:32:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:33:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:33:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:34:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:34:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:35:00 sshguard 12783 Exiting on signal.
Nov 2 06:35:00 sshguard 14139 Now monitoring attacks.
Nov 2 06:35:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:35:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:35:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:36:29 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:36:31 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:36:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:37:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:37:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:38:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:38:29 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:38:31 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:39:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:39:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:39:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:39:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:40:14 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:40:16 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:40:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:40:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:41:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:41:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:42:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:43:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:43:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:44:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:45:14 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:45:16 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:46:54 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:46:56 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:47:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:48:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:48:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:49:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:50:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:50:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:51:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:51:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:51:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:51:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:52:25 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:52:27 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:52:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:53:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:53:44 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:53:46 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:54:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:54:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:54:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:55:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:58:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:58:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:59:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 06:59:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 06:59:59 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:00:01 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:03:14 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:03:16 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:04:14 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:04:16 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:06:31 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:06:33 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:09:15 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:09:17 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:10:15 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:10:16 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:14:19 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:14:21 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:15:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:15:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:16:19 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:16:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:17:04 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:17:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:19:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:19:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:20:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:20:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:21:00 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:21:03 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:22:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:22:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:23:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:23:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:24:02 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:24:04 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:25:03 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:25:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:26:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:26:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:27:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:27:09 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:28:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:28:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:28:33 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:28:36 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:29:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:29:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:29:56 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:29:58 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:30:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:30:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:31:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:31:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:32:20 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:32:22 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:33:05 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:33:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:34:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:34:10 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:35:07 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:35:10 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:36:25 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:36:27 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:38:08 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:38:11 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:39:08 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:39:11 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:40:15 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5e to 00:11:32:a1:cd:5d on igc3
Nov 2 07:40:17 kernel arp: 10.10.10.3 moved from 00:11:32:a1:cd:5d to 00:11:32:a1:cd:5e on igc3
Nov 2 07:42:51 php-fpm 25002 /index.php: Successful login for user 'admin' from: 10.10.10.198 (Local Database)
Nov 2 07:43:18 php-fpm 76039 /diag_reboot.php: Stopping all packages.Any help would be greatly appreciated.
-
What is igc0 there. At 4.09 it lost link. So whatever that's connected to rebooted or maybe there's a cable issue.
It linked back 3s later which started a new mpd process. But it looks like the old process was still running and they are competing.
Do you have any VIPs on the PPPoE interface? https://redmine.pfsense.org/issues/14434
Steve
-
@stephenw10 said in Why did my WAN drop and not reconnect?:
igc0
igc0 links to my ONT (optical network termination unit).
I'm sorry I don't know what you mean by VIP's, is this something my ISP may be able to answer?
-
Ah virtual IP. It's not something I've configured.
-
Ok so igc0 lost link do you know if the ONT rebooted?
Check the chipset version for igc0. Some of the early variants of igc are known to have issues that can present as losing link.
Run:pciconf -lv igc0Steve
-
@stephenw10 said in Why did my WAN drop and not reconnect?:
pciconf -lv igc0
Sorry for the late reply.
Outcome is igc0@pci0:2:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'Ethernet Controller I226-V'
class = network
subclass = ethernetIt's happened 3 times this weekend. WAN drops out, pfsense hasn't locked up and ISP state it's nothing their side.
It's getting to the point where I may have to go back to using my ISP router but I really don't want to.
-
I would try re-assigning WAN to one of the other NICs to make sure it's not just a bad port.
If you can use something that isn't igc I would try that but I imagine all the ports are igc.
That igc NIC shows as rev 4 which should be OK but....
-
@stephenw10 Will give that a shot. Thanks. Will feedback.
-
We seem to have a similar issue.
Here is where I am at: https://forum.netgate.com/topic/185096/pfsense-fails-badly-and-it-is-a-sad-day/1
If you find a solution I am interested.Mike
pciconf -lv igc0
igc0@pci0:86:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x3037
vendor = 'Intel Corporation'
device = 'Ethernet Controller I226-V'
class = network
subclass = ethernetpciconf -lv igc1
igc1@pci0:87:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'Ethernet Controller I226-V'
class = network
subclass = ethernet -
So it stops passing traffic entirely but only on the WAN side?
If you run a packet capture does it show anything when it's in that state?
-
@stephenw10
so it fails dpinger, goes offline. then one of 2 things happen- vpn successfully reconnects with wan still failing. traffic goes out vpn for a while. then vpn fails as well.
- all successful traffic fails.
As for packet capture, been a few months since I did that, if I remember right there were outbound packets with no responses. until I bounced the interface or rebooted the FW
Yes, I tried putting a switch between router and modem. didn't make any difference. same behavior.
tried a bunch of the things that people post as solutions, with no success.Mike
-
Hmm only other thing I'd check is the MAC stats in the sysctls to see if that shows anything coming in at all.
-
@stephenw10
"MAC stats in the sysctls" Guess i need to look that up and do some learning. thanks. -
Check in
sysctl dev.igc.0for example
