WAN interface stops working every few days.
-
You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.
Steve
I'll have to grab that info the next time this behavior occurs, which specific info would I want to grab in this case?
Not sure on that specific AP, it was likely picking up traffic from some random device on my network.
Here's my config related to my PPOE wan if that helps any.
<wan> <if>pppoe0</if> <blockbogons></blockbogons> <descr><![CDATA[WAN01_CenturyLink]]></descr> <alias-address></alias-address> <alias-subnet>32</alias-subnet> <spoofmac></spoofmac> <blockpriv></blockpriv> <enable></enable> <ipaddr>pppoe</ipaddr> </wan> <vlan> <if>igb0</if> <tag>201</tag> <pcp></pcp> <descr><![CDATA[WAN_01_VLAN201]]></descr> <vlanif>igb0.201</vlanif> </vlan> <ppps> <ppp> <ptpid>0</ptpid> <type>pppoe</type> <if>pppoe0</if> <ports>igb0.201</ports> <username><![CDATA[REDACTED@centurylink.net]]></username> <password><![CDATA[REDACTED]]></password> <bandwidth></bandwidth> <mtu></mtu> <mru></mru> <mrru></mrru> </ppp> </ppps> <gateways> <gateway_item> <interface>wan</interface> <gateway>dynamic</gateway> <name>WAN01_CENTURYLINK_PPPOE</name> <weight>1</weight> <ipprotocol>inet</ipprotocol> <descr><![CDATA[Interface WAN01_CENTURYLINK_PPPOE Gateway]]></descr> <monitor>8.8.8.8</monitor> </gateway_item> <defaultgw4>WAN01_CENTURYLINK_PPPOE</defaultgw4> <defaultgw6>-</defaultgw6> </gateways> -
Nothing unusual there.
You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Nothing unusual there.
You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.
Steve
It's unfortunately sometimes occurs more frequently then that. Last event was yesterday around ~1pm and it reoccured a short bit ago around 9:20am today.
I was not able to get the connection back this time by disconnecting and reconnecting the PPOE cconnection, ended up restarting PFsense.
Next step will likely be for me to disable snort for atleast a week or until the issue returns to see if the behaviour reappears.
I'm kind of grasping at straws right now though.....
------------ System logs from time period ---------
Aug 31 09:10:20 snort 67712 [1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.77.227:33798 -> 71.36.122.177:443 Aug 31 09:10:57 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 183.131.3.210:58864 -> 71.36.122.177:1433 Aug 31 09:11:25 snort 67712 [1:2403368:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 35 [Classification: Misc Attack] [Priority: 2] {TCP} 51.161.12.231:32767 -> 71.36.122.177:8545 Aug 31 09:13:13 snort 67712 [1:2403448:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 75 [Classification: Misc Attack] [Priority: 2] {TCP} 89.248.168.157:37856 -> 71.36.122.177:41065 Aug 31 09:14:38 snort 67712 [1:2403458:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 80 [Classification: Misc Attack] [Priority: 2] {TCP} 92.63.197.55:40327 -> 71.36.122.177:3377 Aug 31 09:15:07 snort 67712 [1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.56.238:55872 -> 71.36.122.177:5900 Aug 31 09:16:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.20:57576 -> 71.36.122.177:3345 Aug 31 09:16:14 rc.gateway_alarm 27046 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:506.622ms RTTsd:787.570ms Loss:0%) Aug 31 09:16:14 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:16:14 check_reload_status Restarting ipsec tunnels Aug 31 09:16:14 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:16:14 check_reload_status Reloading filter Aug 31 09:16:15 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:16:15 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:07 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.3:55957 -> 71.36.122.177:3310 Aug 31 09:17:07 snort 67712 [1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291 Aug 31 09:17:07 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291 Aug 31 09:17:22 rc.gateway_alarm 11126 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4120.023ms RTTsd:1799.455ms Loss:22%) Aug 31 09:17:22 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:17:22 check_reload_status Restarting ipsec tunnels Aug 31 09:17:22 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:17:22 check_reload_status Reloading filter Aug 31 09:17:23 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:17:23 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:27 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852 Aug 31 09:17:27 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852 Aug 31 09:17:35 rc.gateway_alarm 61503 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3703.111ms RTTsd:2201.113ms Loss:11%) Aug 31 09:17:35 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:17:35 check_reload_status Restarting ipsec tunnels Aug 31 09:17:35 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:17:35 check_reload_status Reloading filter Aug 31 09:17:36 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:17:36 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:38 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.102:47924 -> 71.36.122.177:26098 Aug 31 09:18:31 snort 67712 [1:2403424:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 63 [Classification: Misc Attack] [Priority: 2] {TCP} 78.108.177.54:26525 -> 71.36.122.177:8080 Aug 31 09:18:32 rc.gateway_alarm 50465 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:0 RTT:310.577ms RTTsd:435.870ms Loss:0%) Aug 31 09:18:32 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:18:32 check_reload_status Restarting ipsec tunnels Aug 31 09:18:32 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:18:32 check_reload_status Reloading filter Aug 31 09:18:33 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:18:34 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:18:57 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.91:45181 -> 71.36.122.177:33355 Aug 31 09:19:52 snort 67712 [1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.113:42826 -> 71.36.122.177:3391 Aug 31 09:20:03 snort 67712 [1:2400005:2773] ET DROP Spamhaus DROP Listed Traffic Inbound group 6 [Classification: Misc Attack] [Priority: 2] {TCP} 103.215.80.70:6000 -> 71.36.122.177:6780 Aug 31 09:20:44 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573 Aug 31 09:20:44 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573 Aug 31 09:22:03 snort 67712 [1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060 Aug 31 09:22:03 snort 67712 [1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060 Aug 31 09:22:27 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433 Aug 31 09:22:29 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433 Aug 31 09:24:01 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.4:55935 -> 71.36.122.177:835 Aug 31 09:24:26 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.8:55838 -> 71.36.122.177:4004 Aug 31 09:26:21 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124 Aug 31 09:26:21 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124 Aug 31 09:27:05 snort 67712 [1:2403406:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 54 [Classification: Misc Attack] [Priority: 2] {TCP} 62.171.161.187:43973 -> 71.36.122.177:81 Aug 31 09:28:11 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606 Aug 31 09:28:11 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606 Aug 31 09:28:47 snort 67712 [1:2403429:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 65 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.212:48824 -> 71.36.122.177:49154 Aug 31 09:28:52 rc.gateway_alarm 69361 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:502.168ms RTTsd:986.015ms Loss:0%) Aug 31 09:28:52 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:28:52 check_reload_status Restarting ipsec tunnels Aug 31 09:28:52 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:28:52 check_reload_status Reloading filter Aug 31 09:28:53 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:28:53 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:28:56 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.53:57620 -> 71.36.122.177:6357 Aug 31 09:29:02 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139 Aug 31 09:29:02 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139 Aug 31 09:29:12 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856 Aug 31 09:29:12 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856 Aug 31 09:29:44 snort 67712 [1:2403419:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 60 [Classification: Misc Attack] [Priority: 2] {UDP} 71.6.158.166:32064 -> 71.36.122.177:389 Aug 31 09:30:04 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867 Aug 31 09:30:04 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867 Aug 31 09:30:14 snort 67712 [1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060 Aug 31 09:30:14 snort 67712 [1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060 Aug 31 09:30:26 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.11:48084 -> 71.36.122.177:10552 Aug 31 09:31:13 rc.gateway_alarm 93277 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4050.647ms RTTsd:1954.397ms Loss:21%) Aug 31 09:31:13 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:31:13 check_reload_status Restarting ipsec tunnels Aug 31 09:31:13 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:31:13 check_reload_status Reloading filter Aug 31 09:31:14 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:31:14 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:31:23 rc.gateway_alarm 78618 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4322.346ms RTTsd:1981.268ms Loss:14%) Aug 31 09:31:23 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:31:23 check_reload_status Restarting ipsec tunnels Aug 31 09:31:23 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:31:23 check_reload_status Reloading filter Aug 31 09:31:24 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:31:24 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:32:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.174:44528 -> 71.36.122.177:33339 Aug 31 09:32:41 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872 Aug 31 09:32:41 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872 Aug 31 09:32:58 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015 Aug 31 09:32:58 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015 Aug 31 09:33:17 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.94:45253 -> 71.36.122.177:33384 Aug 31 09:33:56 snort 67712 [1:2403431:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 66 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.245:44258 -> 71.36.122.177:120 Aug 31 09:34:18 snort 67712 [1:2403436:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 69 [Classification: Misc Attack] [Priority: 2] {TCP} 83.97.20.35:48991 -> 71.36.122.177:6664 Aug 31 09:34:28 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.21:56468 -> 71.36.122.177:22979 Aug 31 09:35:11 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.230:40882 -> 71.36.122.177:3997 Aug 31 09:35:15 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.14:49426 -> 71.36.122.177:26187 Aug 31 09:35:25 snort 67712 [1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.60:53196 -> 71.36.122.177:4184 Aug 31 09:35:38 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139 Aug 31 09:35:38 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139 Aug 31 09:36:18 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 124.114.177.237:10566 -> 71.36.122.177:1433 Aug 31 09:36:35 snort 67712 [1:2403492:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 97 [Classification: Misc Attack] [Priority: 2] {TCP} 106.13.48.122:57394 -> 71.36.122.177:774 Aug 31 09:36:39 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548 Aug 31 09:36:39 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548 Aug 31 09:36:59 snort 67712 [1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.65.74:58855 -> 71.36.122.177:6000 Aug 31 09:37:09 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956 Aug 31 09:37:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956 Aug 31 09:37:11 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.22:56634 -> 71.36.122.177:33046 Aug 31 09:37:31 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547 Aug 31 09:37:31 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547 Aug 31 09:37:33 rc.gateway_alarm 53811 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4054.569ms RTTsd:2049.170ms Loss:21%) Aug 31 09:37:33 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:37:33 check_reload_status Restarting ipsec tunnels Aug 31 09:37:33 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:37:33 check_reload_status Reloading filter Aug 31 09:37:34 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:37:34 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:37:48 snort 67712 [1:2403372:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 37 [Classification: Misc Attack] [Priority: 2] {TCP} 54.36.109.237:50023 -> 71.36.122.177:8443---------- Gateway logs from time period ------------------
Aug 30 13:32:43 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 290791us stddev 369179us loss 0% Aug 31 09:16:14 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 506622us stddev 787570us loss 0% Aug 31 09:17:22 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4120023us stddev 1799455us loss 22% Aug 31 09:17:35 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 3703111us stddev 2201113us loss 11% Aug 31 09:18:32 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 310577us stddev 435870us loss 0% Aug 31 09:28:52 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 502168us stddev 986015us loss 0% Aug 31 09:31:13 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4050647us stddev 1954397us loss 21% Aug 31 09:31:23 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4322346us stddev 1981268us loss 14% Aug 31 09:37:33 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4054569us stddev 2049170us loss 21% Aug 31 09:40:13 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 97.120.6.183 identifier "WAN01_CENTURYLINK_PPPOE " Aug 31 09:40:30 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 507360us stddev 451625us loss 0% Aug 31 09:40:36 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 599186us stddev 671081us loss 22% Aug 31 09:40:46 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1544978us stddev 1669473us loss 11% Aug 31 09:41:13 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1609645us stddev 1562133us loss 21% Aug 31 09:41:18 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 75.164.130.187 identifier "WAN01_CENTURYLINK_PPPOE " Aug 31 09:41:30 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 589734us stddev 844410us loss 14%--- End logs----
I'll need to look closer at the PPP logs the next time this occurs, They were unfortunately flooded out when I restarted pfsense.
I've also been collecting data into Splunk, I'll need to go through that and set up filters when I have time today. -
Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?
-
@stephenw10 said in WAN interface stops working every few days.:
Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?
And it just occurred AGAIN, approx 2 hours later.
Restarted the router another time, This is getting very old and frustrating very fast.
I would love any guidance I can get on next steps.Bullet Points I can think of
- This behavior began a week or so after I switched from A Dell Optiplex 7010 SFF to a Poweredge R210
- Restarting PfSense or the ONT resolve the events when they occur.
- ISP has since replaced ONT.
- Config was imported from the 7010, omitting any package config.
- Have tried 3 different Nics for the Wan IF
- LAN IF is using the onboard Broadcom Nic
- Am not positive on the exact version of PFSense that was on the 7010, I had selected the stable branch and was using whatever it said was up to date.
Could there perhaps be something config related that got corrupted on import and is causing the issues?
------------- TraceRt from router WAN IF -------------------
1 * * * 2 ptld-agw1.inet.qwest.net (207.225.86.145) 1878.017 ms * * 3 * * * 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 1454.335 ms 260.238 ms 249.101 ms 5 74.125.243.177 (74.125.243.177) 158.250 ms 342.457 ms 108.170.245.113 (108.170.245.113) 1406.735 ms 6 * * * 7 * * dns.google (8.8.8.8) 1637.087 ms------------- Ping from router Wan IF ------------------------
PING 8.8.8.8 (8.8.8.8) from 71.36.127.88: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=158.006 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=544.022 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1948.327 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 158.006/883.452/1948.327/769.295 ms------------- TraceRt from router Client IF -------------------
1 ptld-dsl-gw51.ptld.qwest.net (207.225.84.51) 49.551 ms 356.669 ms 1215.833 ms 2 ptld-agw1.inet.qwest.net (207.225.86.145) 443.809 ms 1596.672 ms 1844.559 ms 3 * sea-edge-12.inet.qwest.net (67.14.41.58) 1581.644 ms 14.294 ms 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 22.815 ms 8.851 ms 8.167 ms 5 74.125.243.177 (74.125.243.177) 14.913 ms 108.170.245.97 (108.170.245.97) 8.941 ms 74.125.243.193 (74.125.243.193) 26.185 ms 6 74.125.253.67 (74.125.253.67) 169.668 ms 108.170.233.153 (108.170.233.153) 1183.524 ms 209.85.254.247 (209.85.254.247) 1935.290 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *------------- Ping from router Client IF -----------------------
PING 8.8.8.8 (8.8.8.8) from 192.168.3.1: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=1845.914 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=2216.709 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=3239.383 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1845.914/2434.002/3239.383/589.266 ms----------------- Info from Status > Gateways -------------------
WAN01_CENTURYLINK_PPPOE (default) 207.225.84.51 8.8.4.4 1210.212ms 799.825ms 0.0% Offline Interface WAN01_CENTURYLINK_PPPOE Gateway-------------------- System Logs ---------------------------
(I tried disconnecting and reconnecting around 11:18 at which point it begins to throw Unexpected Protocol IP, Could this hint towards the issue?)Aug 31 09:58:06 check_reload_status Syncing firewall Aug 31 11:03:33 rc.gateway_alarm 87218 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:534.974ms RTTsd:880.397ms Loss:1%) Aug 31 11:03:33 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:03:33 check_reload_status Restarting ipsec tunnels Aug 31 11:03:33 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:03:33 check_reload_status Reloading filter Aug 31 11:03:34 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:03:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:05:53 rc.gateway_alarm 59267 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4196.251ms RTTsd:1499.645ms Loss:21%) Aug 31 11:05:53 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:05:53 check_reload_status Restarting ipsec tunnels Aug 31 11:05:53 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:05:53 check_reload_status Reloading filter Aug 31 11:05:54 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:05:55 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:07:44 php-fpm 73087 /index.php: Successful login for user 'admin' from: 192.168.3.157 (Local Database) Aug 31 11:07:45 rc.gateway_alarm 33853 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3838.708ms RTTsd:1985.755ms Loss:11%) Aug 31 11:07:45 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:07:45 check_reload_status Restarting ipsec tunnels Aug 31 11:07:45 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:07:45 check_reload_status Reloading filter Aug 31 11:07:46 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:07:46 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:19 rc.gateway_alarm 69490 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3395.401ms RTTsd:1821.221ms Loss:21%) Aug 31 11:10:19 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:19 check_reload_status Restarting ipsec tunnels Aug 31 11:10:19 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:19 check_reload_status Reloading filter Aug 31 11:10:20 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:20 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:29 rc.gateway_alarm 20292 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4367.359ms RTTsd:1701.643ms Loss:18%) Aug 31 11:10:29 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:29 check_reload_status Restarting ipsec tunnels Aug 31 11:10:29 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:29 check_reload_status Reloading filter Aug 31 11:10:30 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:31 php-fpm 346 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:32 rc.gateway_alarm 72163 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4591.740ms RTTsd:1589.594ms Loss:21%) Aug 31 11:10:32 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:32 check_reload_status Restarting ipsec tunnels Aug 31 11:10:32 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:32 check_reload_status Reloading filter Aug 31 11:10:33 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:11:01 rc.gateway_alarm 74351 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4430.263ms RTTsd:2115.223ms Loss:16%) Aug 31 11:11:01 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:11:01 check_reload_status Restarting ipsec tunnels Aug 31 11:11:01 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:11:01 check_reload_status Reloading filter Aug 31 11:11:02 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:11:02 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:08 ppp caught fatal signal TERM Aug 31 11:18:08 ppp [wan] IFACE: Close event Aug 31 11:18:08 ppp [wan] IPCP: Close event Aug 31 11:18:08 ppp [wan] IPCP: state change Opened --> Closing Aug 31 11:18:08 ppp [wan] IPCP: SendTerminateReq #4 Aug 31 11:18:08 ppp [wan] IPCP: LayerDown Aug 31 11:18:08 check_reload_status Rewriting resolv.conf Aug 31 11:18:08 ppp [wan] IFACE: Down event Aug 31 11:18:08 ppp [wan] IFACE: Rename interface pppoe0 to pppoe0 Aug 31 11:18:08 ppp [wan] IPV6CP: Close event Aug 31 11:18:08 ppp [wan] IPV6CP: state change Stopped --> Closed Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:09 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan] IPCP: SendTerminateReq #5 Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP **{{{{{{{{{{I deleted 60 or so more repeats of the unexpected Protocol IP error due to character limits in post.}}}}}}}}}}}}}}}** Aug 31 11:18:10 ppp [wan] Bundle: Shutdown Aug 31 11:18:10 ppp [wan_link0] Link: Shutdown Aug 31 11:18:10 ppp process 26141 terminated Aug 31 11:18:13 ppp Multi-link PPP daemon for FreeBSD Aug 31 11:18:13 ppp process 9794 started, version 5.8 (root@pfSense_v2_4_5_amd64-pfSense_v2_4_5-job-04 20:28 17-Dec-2019) Aug 31 11:18:13 ppp web: web is not running Aug 31 11:18:13 ppp [wan] Bundle: Interface ng0 created Aug 31 11:18:13 ppp [wan_link0] Link: OPEN event Aug 31 11:18:13 kernel ng0: changing name to 'pppoe0' Aug 31 11:18:13 ppp [wan_link0] LCP: Open event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan_link0] LCP: LayerStart Aug 31 11:18:13 ppp [wan_link0] PPPoE: Connecting to '' Aug 31 11:18:13 ppp PPPoE: rec'd ACNAME "ptld-dsl-gw51.ptld.qwest.net" Aug 31 11:18:13 ppp [wan_link0] PPPoE: connection successful Aug 31 11:18:13 ppp [wan_link0] Link: UP event Aug 31 11:18:13 ppp [wan_link0] LCP: Up event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Request #9 (Req-Sent) Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigAck #9 Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Ack #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan_link0] LCP: auth: peer wants CHAP, I want nothing Aug 31 11:18:13 ppp [wan_link0] LCP: LayerUp Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd CHALLENGE #244 len: 59 Aug 31 11:18:13 ppp [wan_link0] Name: "JUNOS" Aug 31 11:18:13 ppp [wan_link0] CHAP: Using authname "myerswilliam488@centurylink.net" Aug 31 11:18:13 ppp [wan_link0] CHAP: sending RESPONSE #244 len: 52 Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd SUCCESS #244 len: 4 Aug 31 11:18:13 ppp [wan_link0] LCP: authorization successful Aug 31 11:18:13 ppp [wan_link0] Link: Matched action 'bundle "wan" ""' Aug 31 11:18:13 ppp [wan_link0] Link: Join bundle "wan" Aug 31 11:18:13 ppp [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps Aug 31 11:18:13 ppp [wan] IPCP: Open event Aug 31 11:18:13 ppp [wan] IPCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPCP: LayerStart Aug 31 11:18:13 ppp [wan] IPV6CP: Open event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPV6CP: LayerStart Aug 31 11:18:13 ppp [wan] IPCP: Up event Aug 31 11:18:13 ppp [wan] IPCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPV6CP: Up event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPV6CP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Request #248 (Req-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] 207.225.84.51 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigAck #248 Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] IPCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #2 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Protocol Reject #10 (Opened) Aug 31 11:18:13 ppp [wan_link0] LCP: protocol IPV6CP was rejected Aug 31 11:18:13 ppp [wan] IPV6CP: protocol was rejected by peer Aug 31 11:18:13 ppp [wan] IPV6CP: state change Req-Sent --> Stopped Aug 31 11:18:13 ppp [wan] IPV6CP: LayerFinish Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] 71.36.127.88 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #3 Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan] IPCP: LayerUp Aug 31 11:18:13 ppp [wan] 71.36.127.88 -> 207.225.84.51 Aug 31 11:18:14 check_reload_status rc.newwanip starting pppoe0 Aug 31 11:18:14 ppp [wan] IFACE: Up event Aug 31 11:18:14 ppp [wan] IFACE: Rename interface ng0 to pppoe0 Aug 31 11:18:14 rc.gateway_alarm 11603 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4764.745ms RTTsd:1320.248ms Loss:21%) Aug 31 11:18:14 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:18:14 check_reload_status Restarting ipsec tunnels Aug 31 11:18:14 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:18:14 check_reload_status Reloading filter Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: Info: starting on pppoe0. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: on (IP address: 71.36.127.88) (interface: WAN01_CENTURYLINK[wan]) (real interface: pppoe0). Aug 31 11:18:15 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 207.225.84.51 Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Default gateway setting Interface WAN01_CENTURYLINK_PPPOE Gateway as default. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: IP Address has changed, killing states on former IP Address 71.36.112.131. Aug 31 11:18:16 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:18:17 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:17 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process. Aug 31 11:18:20 dhcpleases kqueue error: unknown Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wancustom''0.cache: 71.36.127.88 Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 31 11:18:22 php-fpm 73087 /rc.newwanip: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:23 php-fpm 73087 /rc.newwanip: Resyncing OpenVPN instances for interface WAN01_CENTURYLINK. Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 64959 Aug 31 11:18:23 kernel ovpns1: link state changed to DOWN Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 kernel ovpns1: link state changed to UP Aug 31 11:18:23 php-fpm 73087 OpenVPN PID written: 98835 Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 check_reload_status rc.newwanip starting ovpns1 Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 91710 Aug 31 11:18:23 kernel ovpns3: link state changed to DOWN Aug 31 11:18:24 kernel ovpns3: link state changed to UP Aug 31 11:18:24 php-fpm 73087 OpenVPN PID written: 20898 Aug 31 11:18:24 php-fpm 73087 /rc.newwanip: Creating rrd update script Aug 31 11:18:24 check_reload_status rc.newwanip starting ovpns3 Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: Info: starting on ovpns1. Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: on (IP address: 192.168.31.1) (interface: []) (real interface: ovpns1). Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:24 check_reload_status Reloading filter Aug 31 11:18:24 php-fpm 346 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.31.1 - Restarting packages. Aug 31 11:18:24 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: Info: starting on ovpns3. Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: on (IP address: 192.168.32.1) (interface: []) (real interface: ovpns3). Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:25 check_reload_status Reloading filter Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.32.1 - Restarting packages. Aug 31 11:18:25 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Restarting/Starting all packages. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service avahi Aug 31 11:18:25 avahi-daemon 71257 Got SIGTERM, quitting. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 71257 avahi-daemon 0.7 exiting. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service avahi Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service nut Aug 31 11:18:25 upsmon 16972 Signal 15: exiting Aug 31 11:18:25 upsd 17558 User local-monitor@::1 logged out from UPS [TrippLite_SMART1500LCD] Aug 31 11:18:25 upsd 17558 mainloop: Interrupted system call Aug 31 11:18:25 upsd 17558 Signal 15: exiting Aug 31 11:18:25 usbhid-ups 17176 Signal 15: exiting Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service nut Aug 31 11:18:25 upsmon 78411 Startup successful Aug 31 11:18:25 usbhid-ups 79004 Startup successful Aug 31 11:18:25 avahi-daemon 75938 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75938 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75938 avahi-daemon 0.7 starting up. Aug 31 11:18:25 avahi-daemon 75938 WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/sftp-ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.4.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.3.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.2.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Network interface enumeration completed. Aug 31 11:18:25 avahi-daemon 75938 Server startup complete. Host name is Camelot.local. Local service cookie is 1381888320. Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_sftp-ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/sftp-ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75027 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75027 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75027 open(/var/run/avahi-daemon//pid): File exists Aug 31 11:18:25 avahi-daemon 75027 Failed to create PID file: File exists Aug 31 11:18:26 php-fpm 73087 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 71.36.112.131 -> 71.36.127.88 - Restarting packages. -
Most of that taken when it was down?
Was something rebooted at some point in that log? When?
-
@stephenw10
The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface
Aug 31 11:18:08 ppp caught fatal signal TERM
I didn't reboot until ~11:28 or so.
This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.Thoughts?
Here are my nuclear options if I can't figure out anything else.
- Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
- Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
- Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).
-
I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC
-
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...No Mac Spoofing
System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLEThe system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive -
Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.
2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.
Steve
-
@gawainxx said in WAN interface stops working every few days.:
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
If your network setup isn't too complicated, this is what I would have done by now.
If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.
Jeff
-
@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?
Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head
.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_
If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)
-
Could a NAT rule for a Nintendo switch cause any issues?
<outbound> <mode>hybrid</mode> <rule> <source> <network>192.168.3.30/32</network> </source> <sourceport></sourceport> <descr><![CDATA[Nindento Switch|Static NAT]]></descr> <target></target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <updated> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </updated> <created> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </created> </rule>I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm
-
No, an outbound NAT rule will not be doing anything.
Traffic shaping is far more likely. Assuming it's anything config related.
Steve
-
Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.
Problem still persists.
There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.
Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?
I'm getting towards the end of my list of ideas and could desperately use some solutions.
I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?
Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xmlI'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)
-
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
-
What version of pfsense??
-
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
