WAN interface stops working every few days.
-
@stephenw10 said in WAN interface stops working every few days.:
Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?
And it just occurred AGAIN, approx 2 hours later.
Restarted the router another time, This is getting very old and frustrating very fast.
I would love any guidance I can get on next steps.Bullet Points I can think of
- This behavior began a week or so after I switched from A Dell Optiplex 7010 SFF to a Poweredge R210
- Restarting PfSense or the ONT resolve the events when they occur.
- ISP has since replaced ONT.
- Config was imported from the 7010, omitting any package config.
- Have tried 3 different Nics for the Wan IF
- LAN IF is using the onboard Broadcom Nic
- Am not positive on the exact version of PFSense that was on the 7010, I had selected the stable branch and was using whatever it said was up to date.
Could there perhaps be something config related that got corrupted on import and is causing the issues?
------------- TraceRt from router WAN IF -------------------
1 * * * 2 ptld-agw1.inet.qwest.net (207.225.86.145) 1878.017 ms * * 3 * * * 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 1454.335 ms 260.238 ms 249.101 ms 5 74.125.243.177 (74.125.243.177) 158.250 ms 342.457 ms 108.170.245.113 (108.170.245.113) 1406.735 ms 6 * * * 7 * * dns.google (8.8.8.8) 1637.087 ms------------- Ping from router Wan IF ------------------------
PING 8.8.8.8 (8.8.8.8) from 71.36.127.88: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=158.006 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=544.022 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1948.327 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 158.006/883.452/1948.327/769.295 ms------------- TraceRt from router Client IF -------------------
1 ptld-dsl-gw51.ptld.qwest.net (207.225.84.51) 49.551 ms 356.669 ms 1215.833 ms 2 ptld-agw1.inet.qwest.net (207.225.86.145) 443.809 ms 1596.672 ms 1844.559 ms 3 * sea-edge-12.inet.qwest.net (67.14.41.58) 1581.644 ms 14.294 ms 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 22.815 ms 8.851 ms 8.167 ms 5 74.125.243.177 (74.125.243.177) 14.913 ms 108.170.245.97 (108.170.245.97) 8.941 ms 74.125.243.193 (74.125.243.193) 26.185 ms 6 74.125.253.67 (74.125.253.67) 169.668 ms 108.170.233.153 (108.170.233.153) 1183.524 ms 209.85.254.247 (209.85.254.247) 1935.290 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *------------- Ping from router Client IF -----------------------
PING 8.8.8.8 (8.8.8.8) from 192.168.3.1: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=1845.914 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=2216.709 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=3239.383 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1845.914/2434.002/3239.383/589.266 ms----------------- Info from Status > Gateways -------------------
WAN01_CENTURYLINK_PPPOE (default) 207.225.84.51 8.8.4.4 1210.212ms 799.825ms 0.0% Offline Interface WAN01_CENTURYLINK_PPPOE Gateway-------------------- System Logs ---------------------------
(I tried disconnecting and reconnecting around 11:18 at which point it begins to throw Unexpected Protocol IP, Could this hint towards the issue?)Aug 31 09:58:06 check_reload_status Syncing firewall Aug 31 11:03:33 rc.gateway_alarm 87218 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:534.974ms RTTsd:880.397ms Loss:1%) Aug 31 11:03:33 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:03:33 check_reload_status Restarting ipsec tunnels Aug 31 11:03:33 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:03:33 check_reload_status Reloading filter Aug 31 11:03:34 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:03:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:05:53 rc.gateway_alarm 59267 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4196.251ms RTTsd:1499.645ms Loss:21%) Aug 31 11:05:53 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:05:53 check_reload_status Restarting ipsec tunnels Aug 31 11:05:53 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:05:53 check_reload_status Reloading filter Aug 31 11:05:54 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:05:55 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:07:44 php-fpm 73087 /index.php: Successful login for user 'admin' from: 192.168.3.157 (Local Database) Aug 31 11:07:45 rc.gateway_alarm 33853 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3838.708ms RTTsd:1985.755ms Loss:11%) Aug 31 11:07:45 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:07:45 check_reload_status Restarting ipsec tunnels Aug 31 11:07:45 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:07:45 check_reload_status Reloading filter Aug 31 11:07:46 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:07:46 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:19 rc.gateway_alarm 69490 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3395.401ms RTTsd:1821.221ms Loss:21%) Aug 31 11:10:19 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:19 check_reload_status Restarting ipsec tunnels Aug 31 11:10:19 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:19 check_reload_status Reloading filter Aug 31 11:10:20 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:20 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:29 rc.gateway_alarm 20292 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4367.359ms RTTsd:1701.643ms Loss:18%) Aug 31 11:10:29 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:29 check_reload_status Restarting ipsec tunnels Aug 31 11:10:29 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:29 check_reload_status Reloading filter Aug 31 11:10:30 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:31 php-fpm 346 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:32 rc.gateway_alarm 72163 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4591.740ms RTTsd:1589.594ms Loss:21%) Aug 31 11:10:32 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:32 check_reload_status Restarting ipsec tunnels Aug 31 11:10:32 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:32 check_reload_status Reloading filter Aug 31 11:10:33 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:11:01 rc.gateway_alarm 74351 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4430.263ms RTTsd:2115.223ms Loss:16%) Aug 31 11:11:01 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:11:01 check_reload_status Restarting ipsec tunnels Aug 31 11:11:01 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:11:01 check_reload_status Reloading filter Aug 31 11:11:02 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:11:02 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:08 ppp caught fatal signal TERM Aug 31 11:18:08 ppp [wan] IFACE: Close event Aug 31 11:18:08 ppp [wan] IPCP: Close event Aug 31 11:18:08 ppp [wan] IPCP: state change Opened --> Closing Aug 31 11:18:08 ppp [wan] IPCP: SendTerminateReq #4 Aug 31 11:18:08 ppp [wan] IPCP: LayerDown Aug 31 11:18:08 check_reload_status Rewriting resolv.conf Aug 31 11:18:08 ppp [wan] IFACE: Down event Aug 31 11:18:08 ppp [wan] IFACE: Rename interface pppoe0 to pppoe0 Aug 31 11:18:08 ppp [wan] IPV6CP: Close event Aug 31 11:18:08 ppp [wan] IPV6CP: state change Stopped --> Closed Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:09 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan] IPCP: SendTerminateReq #5 Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP **{{{{{{{{{{I deleted 60 or so more repeats of the unexpected Protocol IP error due to character limits in post.}}}}}}}}}}}}}}}** Aug 31 11:18:10 ppp [wan] Bundle: Shutdown Aug 31 11:18:10 ppp [wan_link0] Link: Shutdown Aug 31 11:18:10 ppp process 26141 terminated Aug 31 11:18:13 ppp Multi-link PPP daemon for FreeBSD Aug 31 11:18:13 ppp process 9794 started, version 5.8 (root@pfSense_v2_4_5_amd64-pfSense_v2_4_5-job-04 20:28 17-Dec-2019) Aug 31 11:18:13 ppp web: web is not running Aug 31 11:18:13 ppp [wan] Bundle: Interface ng0 created Aug 31 11:18:13 ppp [wan_link0] Link: OPEN event Aug 31 11:18:13 kernel ng0: changing name to 'pppoe0' Aug 31 11:18:13 ppp [wan_link0] LCP: Open event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan_link0] LCP: LayerStart Aug 31 11:18:13 ppp [wan_link0] PPPoE: Connecting to '' Aug 31 11:18:13 ppp PPPoE: rec'd ACNAME "ptld-dsl-gw51.ptld.qwest.net" Aug 31 11:18:13 ppp [wan_link0] PPPoE: connection successful Aug 31 11:18:13 ppp [wan_link0] Link: UP event Aug 31 11:18:13 ppp [wan_link0] LCP: Up event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Request #9 (Req-Sent) Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigAck #9 Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Ack #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan_link0] LCP: auth: peer wants CHAP, I want nothing Aug 31 11:18:13 ppp [wan_link0] LCP: LayerUp Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd CHALLENGE #244 len: 59 Aug 31 11:18:13 ppp [wan_link0] Name: "JUNOS" Aug 31 11:18:13 ppp [wan_link0] CHAP: Using authname "myerswilliam488@centurylink.net" Aug 31 11:18:13 ppp [wan_link0] CHAP: sending RESPONSE #244 len: 52 Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd SUCCESS #244 len: 4 Aug 31 11:18:13 ppp [wan_link0] LCP: authorization successful Aug 31 11:18:13 ppp [wan_link0] Link: Matched action 'bundle "wan" ""' Aug 31 11:18:13 ppp [wan_link0] Link: Join bundle "wan" Aug 31 11:18:13 ppp [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps Aug 31 11:18:13 ppp [wan] IPCP: Open event Aug 31 11:18:13 ppp [wan] IPCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPCP: LayerStart Aug 31 11:18:13 ppp [wan] IPV6CP: Open event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPV6CP: LayerStart Aug 31 11:18:13 ppp [wan] IPCP: Up event Aug 31 11:18:13 ppp [wan] IPCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPV6CP: Up event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPV6CP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Request #248 (Req-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] 207.225.84.51 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigAck #248 Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] IPCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #2 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Protocol Reject #10 (Opened) Aug 31 11:18:13 ppp [wan_link0] LCP: protocol IPV6CP was rejected Aug 31 11:18:13 ppp [wan] IPV6CP: protocol was rejected by peer Aug 31 11:18:13 ppp [wan] IPV6CP: state change Req-Sent --> Stopped Aug 31 11:18:13 ppp [wan] IPV6CP: LayerFinish Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] 71.36.127.88 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #3 Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan] IPCP: LayerUp Aug 31 11:18:13 ppp [wan] 71.36.127.88 -> 207.225.84.51 Aug 31 11:18:14 check_reload_status rc.newwanip starting pppoe0 Aug 31 11:18:14 ppp [wan] IFACE: Up event Aug 31 11:18:14 ppp [wan] IFACE: Rename interface ng0 to pppoe0 Aug 31 11:18:14 rc.gateway_alarm 11603 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4764.745ms RTTsd:1320.248ms Loss:21%) Aug 31 11:18:14 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:18:14 check_reload_status Restarting ipsec tunnels Aug 31 11:18:14 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:18:14 check_reload_status Reloading filter Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: Info: starting on pppoe0. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: on (IP address: 71.36.127.88) (interface: WAN01_CENTURYLINK[wan]) (real interface: pppoe0). Aug 31 11:18:15 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 207.225.84.51 Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Default gateway setting Interface WAN01_CENTURYLINK_PPPOE Gateway as default. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: IP Address has changed, killing states on former IP Address 71.36.112.131. Aug 31 11:18:16 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:18:17 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:17 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process. Aug 31 11:18:20 dhcpleases kqueue error: unknown Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wancustom''0.cache: 71.36.127.88 Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 31 11:18:22 php-fpm 73087 /rc.newwanip: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:23 php-fpm 73087 /rc.newwanip: Resyncing OpenVPN instances for interface WAN01_CENTURYLINK. Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 64959 Aug 31 11:18:23 kernel ovpns1: link state changed to DOWN Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 kernel ovpns1: link state changed to UP Aug 31 11:18:23 php-fpm 73087 OpenVPN PID written: 98835 Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 check_reload_status rc.newwanip starting ovpns1 Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 91710 Aug 31 11:18:23 kernel ovpns3: link state changed to DOWN Aug 31 11:18:24 kernel ovpns3: link state changed to UP Aug 31 11:18:24 php-fpm 73087 OpenVPN PID written: 20898 Aug 31 11:18:24 php-fpm 73087 /rc.newwanip: Creating rrd update script Aug 31 11:18:24 check_reload_status rc.newwanip starting ovpns3 Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: Info: starting on ovpns1. Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: on (IP address: 192.168.31.1) (interface: []) (real interface: ovpns1). Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:24 check_reload_status Reloading filter Aug 31 11:18:24 php-fpm 346 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.31.1 - Restarting packages. Aug 31 11:18:24 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: Info: starting on ovpns3. Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: on (IP address: 192.168.32.1) (interface: []) (real interface: ovpns3). Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:25 check_reload_status Reloading filter Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.32.1 - Restarting packages. Aug 31 11:18:25 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Restarting/Starting all packages. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service avahi Aug 31 11:18:25 avahi-daemon 71257 Got SIGTERM, quitting. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 71257 avahi-daemon 0.7 exiting. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service avahi Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service nut Aug 31 11:18:25 upsmon 16972 Signal 15: exiting Aug 31 11:18:25 upsd 17558 User local-monitor@::1 logged out from UPS [TrippLite_SMART1500LCD] Aug 31 11:18:25 upsd 17558 mainloop: Interrupted system call Aug 31 11:18:25 upsd 17558 Signal 15: exiting Aug 31 11:18:25 usbhid-ups 17176 Signal 15: exiting Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service nut Aug 31 11:18:25 upsmon 78411 Startup successful Aug 31 11:18:25 usbhid-ups 79004 Startup successful Aug 31 11:18:25 avahi-daemon 75938 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75938 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75938 avahi-daemon 0.7 starting up. Aug 31 11:18:25 avahi-daemon 75938 WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/sftp-ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.4.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.3.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.2.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Network interface enumeration completed. Aug 31 11:18:25 avahi-daemon 75938 Server startup complete. Host name is Camelot.local. Local service cookie is 1381888320. Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_sftp-ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/sftp-ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75027 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75027 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75027 open(/var/run/avahi-daemon//pid): File exists Aug 31 11:18:25 avahi-daemon 75027 Failed to create PID file: File exists Aug 31 11:18:26 php-fpm 73087 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 71.36.112.131 -> 71.36.127.88 - Restarting packages. -
Most of that taken when it was down?
Was something rebooted at some point in that log? When?
-
@stephenw10
The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface
Aug 31 11:18:08 ppp caught fatal signal TERM
I didn't reboot until ~11:28 or so.
This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.Thoughts?
Here are my nuclear options if I can't figure out anything else.
- Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
- Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
- Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).
-
I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC
-
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...No Mac Spoofing
System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLEThe system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive -
Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.
2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.
Steve
-
@gawainxx said in WAN interface stops working every few days.:
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
If your network setup isn't too complicated, this is what I would have done by now.
If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.
Jeff
-
@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?
Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head
.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_
If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)
-
Could a NAT rule for a Nintendo switch cause any issues?
<outbound> <mode>hybrid</mode> <rule> <source> <network>192.168.3.30/32</network> </source> <sourceport></sourceport> <descr><![CDATA[Nindento Switch|Static NAT]]></descr> <target></target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <updated> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </updated> <created> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </created> </rule>I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm
-
No, an outbound NAT rule will not be doing anything.
Traffic shaping is far more likely. Assuming it's anything config related.
Steve
-
Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.
Problem still persists.
There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.
Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?
I'm getting towards the end of my list of ideas and could desperately use some solutions.
I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?
Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xmlI'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)
-
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
-
What version of pfsense??
-
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
-
@stephenw10 said in WAN interface stops working every few days.:
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
I'm not using the ISP router for routing or dhcp atm, just handling the vlan tagged traffic to see if it has any influence...
I may have to suffer and try running a double NAT for a week or two though to see if the behaviour persists when ISP router handles traffic. -
@stephenw10 said in WAN interface stops working every few days.:
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
Interesting, I'll need to take a close look at that thread later. The webui does definately take several seconds to load when I initially try to access it while the gateway issues are occuring
-
If you are somehow hitting that still you would see high latency to the firewall itself from a LAN side client everytime you ran Status > Filter reload.
Steve
-
hmm, manually doing a filter reload caused the ping to firewall to jump from 1 to 100ms for a single ping but nothing noteworthy aside from that.
I am however still using the ISP "Modem" (Truly a router) to handle the VLAN tagging, which i set up last night.
-
@gawainxx I already faced this kind of trouble.
The latency starts growing until th Wan interface stops working.
A (not) solution was to turn monitoring IP to of.Then I changed my ISP and both, the trouble and the ISP, desappeared.
