WAN interface stops working every few days.

gawainxx

ISP replaced the ONT and I had been problem free until today when the behaviour appeared again..

I tried to do a tracert and every hop diddnt response and the last 8.8.8.8 had a response time of 1248ms

I was able to restore my connection by going to status>interfaces and then disconnecting and recconecting the WAN PPOE.

Could use some guidance on troubleshooting PPOE issues as well as reccomendations on a scripted workaround to automatically restart it if non responsive after a period of time.

stephenw10

You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?

You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.

Steve

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?

You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.

Steve

I'll have to grab that info the next time this behavior occurs, which specific info would I want to grab in this case?

Not sure on that specific AP, it was likely picking up traffic from some random device on my network.

Here's my config related to my PPOE wan if that helps any.

	<wan>
		<if>pppoe0</if>
		<blockbogons></blockbogons>
		<descr><![CDATA[WAN01_CenturyLink]]></descr>
		<alias-address></alias-address>
		<alias-subnet>32</alias-subnet>
		<spoofmac></spoofmac>
		<blockpriv></blockpriv>
		<enable></enable>
		<ipaddr>pppoe</ipaddr>
	</wan>
	<vlan>
		<if>igb0</if>
		<tag>201</tag>
		<pcp></pcp>
		<descr><![CDATA[WAN_01_VLAN201]]></descr>
		<vlanif>igb0.201</vlanif>
	</vlan>
<ppps>
	<ppp>
		<ptpid>0</ptpid>
		<type>pppoe</type>
		<if>pppoe0</if>
		<ports>igb0.201</ports>
		<username><![CDATA[REDACTED@centurylink.net]]></username>
		<password><![CDATA[REDACTED]]></password>
		<bandwidth></bandwidth>
		<mtu></mtu>
		<mru></mru>
		<mrru></mrru>
	</ppp>
</ppps>
<gateways>
	<gateway_item>
		<interface>wan</interface>
		<gateway>dynamic</gateway>
		<name>WAN01_CENTURYLINK_PPPOE</name>
		<weight>1</weight>
		<ipprotocol>inet</ipprotocol>
		<descr><![CDATA[Interface WAN01_CENTURYLINK_PPPOE Gateway]]></descr>
		<monitor>8.8.8.8</monitor>
	</gateway_item>
	<defaultgw4>WAN01_CENTURYLINK_PPPOE</defaultgw4>
	<defaultgw6>-</defaultgw6>
</gateways>

stephenw10

Nothing unusual there.

You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.

Steve

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

Nothing unusual there.

You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.

Steve

It's unfortunately sometimes occurs more frequently then that. Last event was yesterday around ~1pm and it reoccured a short bit ago around 9:20am today.

I was not able to get the connection back this time by disconnecting and reconnecting the PPOE cconnection, ended up restarting PFsense.

Next step will likely be for me to disable snort for atleast a week or until the issue returns to see if the behaviour reappears.

I'm kind of grasping at straws right now though.....

------------ System logs from time period ---------

Aug 31 09:10:20	snort	67712	[1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.77.227:33798 -> 71.36.122.177:443
Aug 31 09:10:57	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 183.131.3.210:58864 -> 71.36.122.177:1433
Aug 31 09:11:25	snort	67712	[1:2403368:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 35 [Classification: Misc Attack] [Priority: 2] {TCP} 51.161.12.231:32767 -> 71.36.122.177:8545
Aug 31 09:13:13	snort	67712	[1:2403448:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 75 [Classification: Misc Attack] [Priority: 2] {TCP} 89.248.168.157:37856 -> 71.36.122.177:41065
Aug 31 09:14:38	snort	67712	[1:2403458:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 80 [Classification: Misc Attack] [Priority: 2] {TCP} 92.63.197.55:40327 -> 71.36.122.177:3377
Aug 31 09:15:07	snort	67712	[1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.56.238:55872 -> 71.36.122.177:5900
Aug 31 09:16:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.20:57576 -> 71.36.122.177:3345
Aug 31 09:16:14	rc.gateway_alarm	27046	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:506.622ms RTTsd:787.570ms Loss:0%)
Aug 31 09:16:14	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:16:14	check_reload_status		Restarting ipsec tunnels
Aug 31 09:16:14	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:16:14	check_reload_status		Reloading filter
Aug 31 09:16:15	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:16:15	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:17:07	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.3:55957 -> 71.36.122.177:3310
Aug 31 09:17:07	snort	67712	[1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291
Aug 31 09:17:07	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291
Aug 31 09:17:22	rc.gateway_alarm	11126	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4120.023ms RTTsd:1799.455ms Loss:22%)
Aug 31 09:17:22	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:17:22	check_reload_status		Restarting ipsec tunnels
Aug 31 09:17:22	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:17:22	check_reload_status		Reloading filter
Aug 31 09:17:23	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:17:23	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:17:27	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852
Aug 31 09:17:27	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852
Aug 31 09:17:35	rc.gateway_alarm	61503	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3703.111ms RTTsd:2201.113ms Loss:11%)
Aug 31 09:17:35	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:17:35	check_reload_status		Restarting ipsec tunnels
Aug 31 09:17:35	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:17:35	check_reload_status		Reloading filter
Aug 31 09:17:36	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:17:36	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:17:38	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.102:47924 -> 71.36.122.177:26098
Aug 31 09:18:31	snort	67712	[1:2403424:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 63 [Classification: Misc Attack] [Priority: 2] {TCP} 78.108.177.54:26525 -> 71.36.122.177:8080
Aug 31 09:18:32	rc.gateway_alarm	50465	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:0 RTT:310.577ms RTTsd:435.870ms Loss:0%)
Aug 31 09:18:32	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:18:32	check_reload_status		Restarting ipsec tunnels
Aug 31 09:18:32	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:18:32	check_reload_status		Reloading filter
Aug 31 09:18:33	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:18:34	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:18:57	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.91:45181 -> 71.36.122.177:33355
Aug 31 09:19:52	snort	67712	[1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.113:42826 -> 71.36.122.177:3391
Aug 31 09:20:03	snort	67712	[1:2400005:2773] ET DROP Spamhaus DROP Listed Traffic Inbound group 6 [Classification: Misc Attack] [Priority: 2] {TCP} 103.215.80.70:6000 -> 71.36.122.177:6780
Aug 31 09:20:44	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573
Aug 31 09:20:44	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573
Aug 31 09:22:03	snort	67712	[1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060
Aug 31 09:22:03	snort	67712	[1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060
Aug 31 09:22:27	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433
Aug 31 09:22:29	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433
Aug 31 09:24:01	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.4:55935 -> 71.36.122.177:835
Aug 31 09:24:26	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.8:55838 -> 71.36.122.177:4004
Aug 31 09:26:21	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124
Aug 31 09:26:21	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124
Aug 31 09:27:05	snort	67712	[1:2403406:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 54 [Classification: Misc Attack] [Priority: 2] {TCP} 62.171.161.187:43973 -> 71.36.122.177:81
Aug 31 09:28:11	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606
Aug 31 09:28:11	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606
Aug 31 09:28:47	snort	67712	[1:2403429:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 65 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.212:48824 -> 71.36.122.177:49154
Aug 31 09:28:52	rc.gateway_alarm	69361	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:502.168ms RTTsd:986.015ms Loss:0%)
Aug 31 09:28:52	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:28:52	check_reload_status		Restarting ipsec tunnels
Aug 31 09:28:52	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:28:52	check_reload_status		Reloading filter
Aug 31 09:28:53	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:28:53	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:28:56	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.53:57620 -> 71.36.122.177:6357
Aug 31 09:29:02	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139
Aug 31 09:29:02	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139
Aug 31 09:29:12	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856
Aug 31 09:29:12	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856
Aug 31 09:29:44	snort	67712	[1:2403419:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 60 [Classification: Misc Attack] [Priority: 2] {UDP} 71.6.158.166:32064 -> 71.36.122.177:389
Aug 31 09:30:04	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867
Aug 31 09:30:04	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867
Aug 31 09:30:14	snort	67712	[1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060
Aug 31 09:30:14	snort	67712	[1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060
Aug 31 09:30:26	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.11:48084 -> 71.36.122.177:10552
Aug 31 09:31:13	rc.gateway_alarm	93277	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4050.647ms RTTsd:1954.397ms Loss:21%)
Aug 31 09:31:13	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:31:13	check_reload_status		Restarting ipsec tunnels
Aug 31 09:31:13	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:31:13	check_reload_status		Reloading filter
Aug 31 09:31:14	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:31:14	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:31:23	rc.gateway_alarm	78618	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4322.346ms RTTsd:1981.268ms Loss:14%)
Aug 31 09:31:23	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:31:23	check_reload_status		Restarting ipsec tunnels
Aug 31 09:31:23	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:31:23	check_reload_status		Reloading filter
Aug 31 09:31:24	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:31:24	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:32:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.174:44528 -> 71.36.122.177:33339
Aug 31 09:32:41	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872
Aug 31 09:32:41	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872
Aug 31 09:32:58	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015
Aug 31 09:32:58	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015
Aug 31 09:33:17	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.94:45253 -> 71.36.122.177:33384
Aug 31 09:33:56	snort	67712	[1:2403431:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 66 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.245:44258 -> 71.36.122.177:120
Aug 31 09:34:18	snort	67712	[1:2403436:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 69 [Classification: Misc Attack] [Priority: 2] {TCP} 83.97.20.35:48991 -> 71.36.122.177:6664
Aug 31 09:34:28	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.21:56468 -> 71.36.122.177:22979
Aug 31 09:35:11	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.230:40882 -> 71.36.122.177:3997
Aug 31 09:35:15	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.14:49426 -> 71.36.122.177:26187
Aug 31 09:35:25	snort	67712	[1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.60:53196 -> 71.36.122.177:4184
Aug 31 09:35:38	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139
Aug 31 09:35:38	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139
Aug 31 09:36:18	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 124.114.177.237:10566 -> 71.36.122.177:1433
Aug 31 09:36:35	snort	67712	[1:2403492:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 97 [Classification: Misc Attack] [Priority: 2] {TCP} 106.13.48.122:57394 -> 71.36.122.177:774
Aug 31 09:36:39	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548
Aug 31 09:36:39	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548
Aug 31 09:36:59	snort	67712	[1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.65.74:58855 -> 71.36.122.177:6000
Aug 31 09:37:09	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956
Aug 31 09:37:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956
Aug 31 09:37:11	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.22:56634 -> 71.36.122.177:33046
Aug 31 09:37:31	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547
Aug 31 09:37:31	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547
Aug 31 09:37:33	rc.gateway_alarm	53811	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4054.569ms RTTsd:2049.170ms Loss:21%)
Aug 31 09:37:33	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 09:37:33	check_reload_status		Restarting ipsec tunnels
Aug 31 09:37:33	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 09:37:33	check_reload_status		Reloading filter
Aug 31 09:37:34	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 09:37:34	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 09:37:48	snort	67712	[1:2403372:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 37 [Classification: Misc Attack] [Priority: 2] {TCP} 54.36.109.237:50023 -> 71.36.122.177:8443

---------- Gateway logs from time period ------------------

Aug 30 13:32:43	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 290791us stddev 369179us loss 0%
Aug 31 09:16:14	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 506622us stddev 787570us loss 0%
Aug 31 09:17:22	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4120023us stddev 1799455us loss 22%
Aug 31 09:17:35	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 3703111us stddev 2201113us loss 11%
Aug 31 09:18:32	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 310577us stddev 435870us loss 0%
Aug 31 09:28:52	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 502168us stddev 986015us loss 0%
Aug 31 09:31:13	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4050647us stddev 1954397us loss 21%
Aug 31 09:31:23	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4322346us stddev 1981268us loss 14%
Aug 31 09:37:33	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4054569us stddev 2049170us loss 21%
Aug 31 09:40:13	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 97.120.6.183 identifier "WAN01_CENTURYLINK_PPPOE "
Aug 31 09:40:30	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 507360us stddev 451625us loss 0%
Aug 31 09:40:36	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 599186us stddev 671081us loss 22%
Aug 31 09:40:46	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1544978us stddev 1669473us loss 11%
Aug 31 09:41:13	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1609645us stddev 1562133us loss 21%
Aug 31 09:41:18	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 75.164.130.187 identifier "WAN01_CENTURYLINK_PPPOE "
Aug 31 09:41:30	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 589734us stddev 844410us loss 14%

--- End logs----

I'll need to look closer at the PPP logs the next time this occurs, They were unfortunately flooded out when I restarted pfsense.
I've also been collecting data into Splunk, I'll need to go through that and set up filters when I have time today.

stephenw10

Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?

And it just occurred AGAIN, approx 2 hours later.
Restarted the router another time, This is getting very old and frustrating very fast.
I would love any guidance I can get on next steps.

Bullet Points I can think of

This behavior began a week or so after I switched from A Dell Optiplex 7010 SFF to a Poweredge R210
Restarting PfSense or the ONT resolve the events when they occur.
ISP has since replaced ONT.
Config was imported from the 7010, omitting any package config.
Have tried 3 different Nics for the Wan IF
LAN IF is using the onboard Broadcom Nic
Am not positive on the exact version of PFSense that was on the 7010, I had selected the stable branch and was using whatever it said was up to date.

Could there perhaps be something config related that got corrupted on import and is causing the issues?

------------- TraceRt from router WAN IF -------------------

 1  * * *
 2  ptld-agw1.inet.qwest.net (207.225.86.145)  1878.017 ms * *
 3  * * *
 4  63-158-222-114.dia.static.qwest.net (63.158.222.114)  1454.335 ms  260.238 ms  249.101 ms
 5  74.125.243.177 (74.125.243.177)  158.250 ms  342.457 ms
    108.170.245.113 (108.170.245.113)  1406.735 ms
 6  * * *
 7  * * dns.google (8.8.8.8)  1637.087 ms

------------- Ping from router Wan IF ------------------------

PING 8.8.8.8 (8.8.8.8) from 71.36.127.88: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=158.006 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=544.022 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1948.327 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 158.006/883.452/1948.327/769.295 ms

------------- TraceRt from router Client IF -------------------

1  ptld-dsl-gw51.ptld.qwest.net (207.225.84.51)  49.551 ms  356.669 ms  1215.833 ms
 2  ptld-agw1.inet.qwest.net (207.225.86.145)  443.809 ms  1596.672 ms  1844.559 ms
 3  * sea-edge-12.inet.qwest.net (67.14.41.58)  1581.644 ms  14.294 ms
 4  63-158-222-114.dia.static.qwest.net (63.158.222.114)  22.815 ms  8.851 ms  8.167 ms
 5  74.125.243.177 (74.125.243.177)  14.913 ms
    108.170.245.97 (108.170.245.97)  8.941 ms
    74.125.243.193 (74.125.243.193)  26.185 ms
 6  74.125.253.67 (74.125.253.67)  169.668 ms
    108.170.233.153 (108.170.233.153)  1183.524 ms
    209.85.254.247 (209.85.254.247)  1935.290 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

------------- Ping from router Client IF -----------------------

PING 8.8.8.8 (8.8.8.8) from 192.168.3.1: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=1845.914 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=2216.709 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=3239.383 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1845.914/2434.002/3239.383/589.266 ms

----------------- Info from Status > Gateways -------------------

WAN01_CENTURYLINK_PPPOE (default)	207.225.84.51	8.8.4.4	1210.212ms	799.825ms	0.0%	Offline	Interface WAN01_CENTURYLINK_PPPOE Gateway

-------------------- System Logs ---------------------------
(I tried disconnecting and reconnecting around 11:18 at which point it begins to throw Unexpected Protocol IP, Could this hint towards the issue?)

Aug 31 09:58:06	check_reload_status		Syncing firewall
Aug 31 11:03:33	rc.gateway_alarm	87218	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:534.974ms RTTsd:880.397ms Loss:1%)
Aug 31 11:03:33	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:03:33	check_reload_status		Restarting ipsec tunnels
Aug 31 11:03:33	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:03:33	check_reload_status		Reloading filter
Aug 31 11:03:34	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:03:34	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:05:53	rc.gateway_alarm	59267	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4196.251ms RTTsd:1499.645ms Loss:21%)
Aug 31 11:05:53	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:05:53	check_reload_status		Restarting ipsec tunnels
Aug 31 11:05:53	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:05:53	check_reload_status		Reloading filter
Aug 31 11:05:54	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:05:55	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:07:44	php-fpm	73087	/index.php: Successful login for user 'admin' from: 192.168.3.157 (Local Database)
Aug 31 11:07:45	rc.gateway_alarm	33853	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3838.708ms RTTsd:1985.755ms Loss:11%)
Aug 31 11:07:45	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:07:45	check_reload_status		Restarting ipsec tunnels
Aug 31 11:07:45	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:07:45	check_reload_status		Reloading filter
Aug 31 11:07:46	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:07:46	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:10:19	rc.gateway_alarm	69490	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3395.401ms RTTsd:1821.221ms Loss:21%)
Aug 31 11:10:19	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:10:19	check_reload_status		Restarting ipsec tunnels
Aug 31 11:10:19	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:10:19	check_reload_status		Reloading filter
Aug 31 11:10:20	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:10:20	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:10:29	rc.gateway_alarm	20292	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4367.359ms RTTsd:1701.643ms Loss:18%)
Aug 31 11:10:29	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:10:29	check_reload_status		Restarting ipsec tunnels
Aug 31 11:10:29	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:10:29	check_reload_status		Reloading filter
Aug 31 11:10:30	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:10:31	php-fpm	346	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:10:32	rc.gateway_alarm	72163	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4591.740ms RTTsd:1589.594ms Loss:21%)
Aug 31 11:10:32	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:10:32	check_reload_status		Restarting ipsec tunnels
Aug 31 11:10:32	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:10:32	check_reload_status		Reloading filter
Aug 31 11:10:33	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:10:34	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:11:01	rc.gateway_alarm	74351	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4430.263ms RTTsd:2115.223ms Loss:16%)
Aug 31 11:11:01	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:11:01	check_reload_status		Restarting ipsec tunnels
Aug 31 11:11:01	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:11:01	check_reload_status		Reloading filter
Aug 31 11:11:02	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:11:02	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:18:08	ppp		caught fatal signal TERM
Aug 31 11:18:08	ppp		[wan] IFACE: Close event
Aug 31 11:18:08	ppp		[wan] IPCP: Close event
Aug 31 11:18:08	ppp		[wan] IPCP: state change Opened --> Closing
Aug 31 11:18:08	ppp		[wan] IPCP: SendTerminateReq #4
Aug 31 11:18:08	ppp		[wan] IPCP: LayerDown
Aug 31 11:18:08	check_reload_status		Rewriting resolv.conf
Aug 31 11:18:08	ppp		[wan] IFACE: Down event
Aug 31 11:18:08	ppp		[wan] IFACE: Rename interface pppoe0 to pppoe0
Aug 31 11:18:08	ppp		[wan] IPV6CP: Close event
Aug 31 11:18:08	ppp		[wan] IPV6CP: state change Stopped --> Closed
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:09	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan] IPCP: SendTerminateReq #5
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
       **{{{{{{{{{{I deleted 60 or so more repeats of the unexpected Protocol IP error due to character limits in post.}}}}}}}}}}}}}}}**
Aug 31 11:18:10	ppp		[wan] Bundle: Shutdown
Aug 31 11:18:10	ppp		[wan_link0] Link: Shutdown
Aug 31 11:18:10	ppp		process 26141 terminated
Aug 31 11:18:13	ppp		Multi-link PPP daemon for FreeBSD
Aug 31 11:18:13	ppp		process 9794 started, version 5.8 (root@pfSense_v2_4_5_amd64-pfSense_v2_4_5-job-04 20:28 17-Dec-2019)
Aug 31 11:18:13	ppp		web: web is not running
Aug 31 11:18:13	ppp		[wan] Bundle: Interface ng0 created
Aug 31 11:18:13	ppp		[wan_link0] Link: OPEN event
Aug 31 11:18:13	kernel		ng0: changing name to 'pppoe0'
Aug 31 11:18:13	ppp		[wan_link0] LCP: Open event
Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Initial --> Starting
Aug 31 11:18:13	ppp		[wan_link0] LCP: LayerStart
Aug 31 11:18:13	ppp		[wan_link0] PPPoE: Connecting to ''
Aug 31 11:18:13	ppp		PPPoE: rec'd ACNAME "ptld-dsl-gw51.ptld.qwest.net"
Aug 31 11:18:13	ppp		[wan_link0] PPPoE: connection successful
Aug 31 11:18:13	ppp		[wan_link0] Link: UP event
Aug 31 11:18:13	ppp		[wan_link0] LCP: Up event
Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Starting --> Req-Sent
Aug 31 11:18:13	ppp		[wan_link0] LCP: SendConfigReq #1
Aug 31 11:18:13	ppp		[wan_link0] PROTOCOMP
Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x2004df36
Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Configure Request #9 (Req-Sent)
Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
Aug 31 11:18:13	ppp		[wan_link0] AUTHPROTO CHAP MD5
Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x08202657
Aug 31 11:18:13	ppp		[wan_link0] LCP: SendConfigAck #9
Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
Aug 31 11:18:13	ppp		[wan_link0] AUTHPROTO CHAP MD5
Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x08202657
Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Req-Sent --> Ack-Sent
Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Configure Ack #1 (Ack-Sent)
Aug 31 11:18:13	ppp		[wan_link0] PROTOCOMP
Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x2004df36
Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Ack-Sent --> Opened
Aug 31 11:18:13	ppp		[wan_link0] LCP: auth: peer wants CHAP, I want nothing
Aug 31 11:18:13	ppp		[wan_link0] LCP: LayerUp
Aug 31 11:18:13	ppp		[wan_link0] CHAP: rec'd CHALLENGE #244 len: 59
Aug 31 11:18:13	ppp		[wan_link0] Name: "JUNOS"
Aug 31 11:18:13	ppp		[wan_link0] CHAP: Using authname "myerswilliam488@centurylink.net"
Aug 31 11:18:13	ppp		[wan_link0] CHAP: sending RESPONSE #244 len: 52
Aug 31 11:18:13	ppp		[wan_link0] CHAP: rec'd SUCCESS #244 len: 4
Aug 31 11:18:13	ppp		[wan_link0] LCP: authorization successful
Aug 31 11:18:13	ppp		[wan_link0] Link: Matched action 'bundle "wan" ""'
Aug 31 11:18:13	ppp		[wan_link0] Link: Join bundle "wan"
Aug 31 11:18:13	ppp		[wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
Aug 31 11:18:13	ppp		[wan] IPCP: Open event
Aug 31 11:18:13	ppp		[wan] IPCP: state change Initial --> Starting
Aug 31 11:18:13	ppp		[wan] IPCP: LayerStart
Aug 31 11:18:13	ppp		[wan] IPV6CP: Open event
Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Initial --> Starting
Aug 31 11:18:13	ppp		[wan] IPV6CP: LayerStart
Aug 31 11:18:13	ppp		[wan] IPCP: Up event
Aug 31 11:18:13	ppp		[wan] IPCP: state change Starting --> Req-Sent
Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #1
Aug 31 11:18:13	ppp		[wan] IPADDR 0.0.0.0
Aug 31 11:18:13	ppp		[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Aug 31 11:18:13	ppp		[wan] IPV6CP: Up event
Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Starting --> Req-Sent
Aug 31 11:18:13	ppp		[wan] IPV6CP: SendConfigReq #1
Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Request #248 (Req-Sent)
Aug 31 11:18:13	ppp		[wan] IPADDR 207.225.84.51
Aug 31 11:18:13	ppp		[wan] 207.225.84.51 is OK
Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigAck #248
Aug 31 11:18:13	ppp		[wan] IPADDR 207.225.84.51
Aug 31 11:18:13	ppp		[wan] IPCP: state change Req-Sent --> Ack-Sent
Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
Aug 31 11:18:13	ppp		[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #2
Aug 31 11:18:13	ppp		[wan] IPADDR 0.0.0.0
Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Protocol Reject #10 (Opened)
Aug 31 11:18:13	ppp		[wan_link0] LCP: protocol IPV6CP was rejected
Aug 31 11:18:13	ppp		[wan] IPV6CP: protocol was rejected by peer
Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Req-Sent --> Stopped
Aug 31 11:18:13	ppp		[wan] IPV6CP: LayerFinish
Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
Aug 31 11:18:13	ppp		[wan] 71.36.127.88 is OK
Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #3
Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
Aug 31 11:18:13	ppp		[wan] IPCP: state change Ack-Sent --> Opened
Aug 31 11:18:13	ppp		[wan] IPCP: LayerUp
Aug 31 11:18:13	ppp		[wan] 71.36.127.88 -> 207.225.84.51
Aug 31 11:18:14	check_reload_status		rc.newwanip starting pppoe0
Aug 31 11:18:14	ppp		[wan] IFACE: Up event
Aug 31 11:18:14	ppp		[wan] IFACE: Rename interface ng0 to pppoe0
Aug 31 11:18:14	rc.gateway_alarm	11603	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4764.745ms RTTsd:1320.248ms Loss:21%)
Aug 31 11:18:14	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
Aug 31 11:18:14	check_reload_status		Restarting ipsec tunnels
Aug 31 11:18:14	check_reload_status		Restarting OpenVPN tunnels/interfaces
Aug 31 11:18:14	check_reload_status		Reloading filter
Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: rc.newwanip: on (IP address: 71.36.127.88) (interface: WAN01_CENTURYLINK[wan]) (real interface: pppoe0).
Aug 31 11:18:15	dhcpleases		/etc/hosts changed size from original!
Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 207.225.84.51
Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: Default gateway setting Interface WAN01_CENTURYLINK_PPPOE Gateway as default.
Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: IP Address has changed, killing states on former IP Address 71.36.112.131.
Aug 31 11:18:16	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
Aug 31 11:18:17	dhcpleases		/etc/hosts changed size from original!
Aug 31 11:18:17	dhcpleases		Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
Aug 31 11:18:20	dhcpleases		kqueue error: unknown
Aug 31 11:18:22	php-fpm	346	/rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wancustom''0.cache: 71.36.127.88
Aug 31 11:18:22	php-fpm	346	/rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully!
Aug 31 11:18:22	php-fpm	73087	/rc.newwanip: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 31 11:18:23	php-fpm	73087	/rc.newwanip: Resyncing OpenVPN instances for interface WAN01_CENTURYLINK.
Aug 31 11:18:23	php-fpm	73087	OpenVPN terminate old pid: 64959
Aug 31 11:18:23	kernel		ovpns1: link state changed to DOWN
Aug 31 11:18:23	check_reload_status		Reloading filter
Aug 31 11:18:23	kernel		ovpns1: link state changed to UP
Aug 31 11:18:23	php-fpm	73087	OpenVPN PID written: 98835
Aug 31 11:18:23	check_reload_status		Reloading filter
Aug 31 11:18:23	check_reload_status		rc.newwanip starting ovpns1
Aug 31 11:18:23	php-fpm	73087	OpenVPN terminate old pid: 91710
Aug 31 11:18:23	kernel		ovpns3: link state changed to DOWN
Aug 31 11:18:24	kernel		ovpns3: link state changed to UP
Aug 31 11:18:24	php-fpm	73087	OpenVPN PID written: 20898
Aug 31 11:18:24	php-fpm	73087	/rc.newwanip: Creating rrd update script
Aug 31 11:18:24	check_reload_status		rc.newwanip starting ovpns3
Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip: Info: starting on ovpns1.
Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip: on (IP address: 192.168.31.1) (interface: []) (real interface: ovpns1).
Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip called with empty interface.
Aug 31 11:18:24	check_reload_status		Reloading filter
Aug 31 11:18:24	php-fpm	346	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.31.1 - Restarting packages.
Aug 31 11:18:24	check_reload_status		Starting packages
Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip: Info: starting on ovpns3.
Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip: on (IP address: 192.168.32.1) (interface: []) (real interface: ovpns3).
Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip called with empty interface.
Aug 31 11:18:25	check_reload_status		Reloading filter
Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.32.1 - Restarting packages.
Aug 31 11:18:25	check_reload_status		Starting packages
Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Restarting/Starting all packages.
Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Stopping service avahi
Aug 31 11:18:25	avahi-daemon	71257	Got SIGTERM, quitting.
Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1.
Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1.
Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1.
Aug 31 11:18:25	avahi-daemon	71257	avahi-daemon 0.7 exiting.
Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Starting service avahi
Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Stopping service nut
Aug 31 11:18:25	upsmon	16972	Signal 15: exiting
Aug 31 11:18:25	upsd	17558	User local-monitor@::1 logged out from UPS [TrippLite_SMART1500LCD]
Aug 31 11:18:25	upsd	17558	mainloop: Interrupted system call
Aug 31 11:18:25	upsd	17558	Signal 15: exiting
Aug 31 11:18:25	usbhid-ups	17176	Signal 15: exiting
Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Starting service nut
Aug 31 11:18:25	upsmon	78411	Startup successful
Aug 31 11:18:25	usbhid-ups	79004	Startup successful
Aug 31 11:18:25	avahi-daemon	75938	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Aug 31 11:18:25	avahi-daemon	75938	Successfully dropped root privileges.
Aug 31 11:18:25	avahi-daemon	75938	avahi-daemon 0.7 starting up.
Aug 31 11:18:25	avahi-daemon	75938	WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Aug 31 11:18:25	avahi-daemon	75938	Loading service file /usr/local/etc/avahi/services/sftp-ssh.service.
Aug 31 11:18:25	avahi-daemon	75938	Loading service file /usr/local/etc/avahi/services/ssh.service.
Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1.
Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.4.IPv4 for mDNS.
Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1.
Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.3.IPv4 for mDNS.
Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1.
Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.2.IPv4 for mDNS.
Aug 31 11:18:25	avahi-daemon	75938	Network interface enumeration completed.
Aug 31 11:18:25	avahi-daemon	75938	Server startup complete. Host name is Camelot.local. Local service cookie is 1381888320.
Aug 31 11:18:25	avahi-daemon	75938	Failed to add service 'Camelot' of type '_ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/ssh.service): Not permitted
Aug 31 11:18:25	avahi-daemon	75938	Failed to add service 'Camelot' of type '_sftp-ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/sftp-ssh.service): Not permitted
Aug 31 11:18:25	avahi-daemon	75027	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Aug 31 11:18:25	avahi-daemon	75027	Successfully dropped root privileges.
Aug 31 11:18:25	avahi-daemon	75027	open(/var/run/avahi-daemon//pid): File exists
Aug 31 11:18:25	avahi-daemon	75027	Failed to create PID file: File exists
Aug 31 11:18:26	php-fpm	73087	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 71.36.112.131 -> 71.36.127.88 - Restarting packages.

stephenw10

Most of that taken when it was down?

Was something rebooted at some point in that log? When?

gawainxx

@stephenw10
The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.

About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface

Aug 31 11:18:08 ppp caught fatal signal TERM

I didn't reboot until ~11:28 or so.

This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.

Thoughts?

Here are my nuclear options if I can't figure out anything else.

Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).

gawainxx

I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC

stephenw10

Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

You are running 2.4.5p1 right?

Steve

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

You are running 2.4.5p1 right?

Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...

No Mac Spoofing

System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLE

The system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive

stephenw10

Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.

2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.

Steve

akuma1x

@gawainxx said in WAN interface stops working every few days.:

Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?

If your network setup isn't too complicated, this is what I would have done by now.

If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.

Jeff

gawainxx

@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?

I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?

Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head

.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_

If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)

gawainxx

@gawainxx

Could a NAT rule for a Nintendo switch cause any issues?

	<outbound>
		<mode>hybrid</mode>
		<rule>
			<source>
				<network>192.168.3.30/32</network>
			</source>
			<sourceport></sourceport>
			<descr><![CDATA[Nindento Switch|Static NAT]]></descr>
			<target></target>
			<targetip></targetip>
			<targetip_subnet></targetip_subnet>
			<interface>wan</interface>
			<poolopts></poolopts>
			<source_hash_key></source_hash_key>
			<staticnatport></staticnatport>
			<destination>
				<any></any>
			</destination>
			<updated>
				<time>1589685349</time>
				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
			</updated>
			<created>
				<time>1589685349</time>
				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
			</created>
		</rule>

I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm

stephenw10

No, an outbound NAT rule will not be doing anything.

Traffic shaping is far more likely. Assuming it's anything config related.

Steve

gawainxx

Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.

Problem still persists.

There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.

Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?

I'm getting towards the end of my list of ideas and could desperately use some solutions.

I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?

Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xml

I'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)

stephenw10

The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.

If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.

Steve

Cool_Corona

What version of pfsense??