WAN interface stops working every few days.
-
I've been having issues where my WAN interface will crap out after a few days or so.
The behaviour is that I will have very high packet % as well as latency of 7600+MS
already established connections (such as splashtop Remote desktop) seem unphased but any new connections and general net traffic is doa.Rebooting pfsense or power cycling the CenturyLink provided ONT (optical network terminal) both resolve the issue for a day or so.
I've tried disabling and re-enabling the wan interface as well as unplugging and plugging the ethernet cable for WAN but those do not resolve the issue when it appears.What are some likely causes for this?
-
Try using Packet Capture, to see what's happening.
-
What would I want to focus on for the packet capture, WAN?
also, it seems like Unplugging the WAN cable for a minute or so and plugging it back in also resolves the high latency and drops.
-
I've found this in the gateway logs if it helps any. Also Guess I'll set a syslog server back up later today.
Aug 8 11:22:02 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr [Redacted] bind_addr [Redacted] bidentifier "WAN01_CENTURYLINK_PPPOE " Aug 8 13:44:52 dpinger WAN01_CENTURYLINK_PPPOE [Redacted]: Alarm latency 544019us stddev 1364744us loss 10% -
Well, you could start with what happens when you try disconnecting/reconnection the WAN cable or disabling/re-enabling the interface. I also see it's OK after waiting a minute before plugging the cable in. You could compare the differences with not waiting. That sort of thing. About 1.5 years ago, I had a problem with IPv6 on my ISP. By using Wireshark, when pfSense booted up, I was able to identify the failing equipment, by name, at my ISPs local office. After I got that resolved, I saved a normal DHCP & DHCPv6-PD sequence, so that I'd always have something to refer to, should a problem happen again. Since I was rebooting pfsense, I couldn't use Packet Capture, so I used a managed switch, configured as a data tap,with my notebook running Wireshark.
-
This post is deleted! -
@bcruze
These issues began after I swapped my pfsense box from an optiplex 7010 to a Dell Poweredge R210 II. it was working without any issue prior.As of 2 days ago this behaviour is now occuring every 12 hours or so.
As of yesterday, I've already tried tweaking the system tunables per some suggestions for PPOE interfaces as well as BCE adapters.
I'm going to try buying an intel gigabit nic and see if it's an issue with the broadcom onboard adapter.
It also seems that manually setting my PPOE connection as offline, applying settings then going back and re-enabling it temporarily resolves the issue when it occurs whereas the monitoring service isn't managing to re-establish the connection.
-
What NICs were in the older box?
I assume this is a PPPoE connection from your logs, unless that's another gateway?
You could certainly try setting the gateway monitoring target to something different. Be sure it's actually the WAN and not just the target.
Carrier grade NAT should be all at the ISP if they are using that. I'm not sure which issue you're referring to @bcruze.
Steve
-
I'm not certain which was wan/LAN but I had two NICs, an onboard intel gigabit and a broadcom 5722.
I still have the 5722 floating around so I'm going to see if I can adapt a full length PCIE slot bracket onto it
I've already tried setting the gateway monitoring target to 8.8.8.8
https://techtilt.com/fix-for-pfsense-keeps-dropping-wan-intermittently-randomI also tried adjusting a number of settings in tunables.
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html
net.isr.dispatch=deferred
kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0P.S. It doesn't look like syslogging is immediately available because i let my Splunk trial license lapse... I just applied the 500MB free license but it'll take a month or a reinstall before I can access the data again.
-
Installed the 5722 going to see how it handles the PPPOe... Diddnt have a correct full length bracket so I got creative to prevent it from getting knocked out of the slot and causing a short. https://imgur.com/gallery/dbfYaLi
-
Still occurring unfortunately, would I need to run a packet capture 24/7 until an issue occurs or would there be another route?
-
That may not tell you much anyway.
It's curious that rebooting the ONT corrects the issue but unplugging the cable does not. Those should be similar from pfSense's view. Obviously one resets the upstream connection too but if that were an issue then rebooting pfSense alone would not correct it.
A short pcap made whilst the connection is bad might show something. Bad packets etc.
It might need to be on the PPPoE parent interface though.
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
That may not tell you much anyway.
It's curious that rebooting the ONT corrects the issue but unplugging the cable does not. Those should be similar from pfSense's view. Obviously one resets the upstream connection too but if that were an issue then rebooting pfSense alone would not correct it.
A short pcap made whilst the connection is bad might show something. Bad packets etc.
It might need to be on the PPPoE parent interface though.
Steve
Three things have been observed to correct the issue so far.
Rebooting pfsense
Disconnecting the WAN if's ethernet cable for ~15 seconds then plugging it back in.
Power Cycling the ONTpacket loss and latency skyrockets during these events.
I'm going to do a packet capture as well as take a close look at the PPPOE traffic the next time this happens. I'm curious to see if my WAN IP changes as well as what disabling and re-enabling PPPOE does.This issue began approximately 1 week after I had replaced my optiplex 7010SFF PFsense instance for the R210 II.
There are two other things in the same timeframe which "may may potentially attribute but I'd be surprised if they were the issue"- Minor heat wave where temps were in the upper 90's for a few days.\
- Unmounted ONT to physically inspect what type of optical cable it uses, It may be remotely possible that I somehow pinched the cable when returning the ONT back into it's cradle? I'm not certain whether that would manifest with these symptoms though. aside from the every 12-36 hour events pings, latency and packet loss are on par for gigabit.
-
I changed out the Broadcom NIC for an Intel one and I really, really hope this issues goes away with it.
I did a packet capture on the WAN interface and see a lot of ttl timeouts, TCP resets or unacknowledged acks. Existing socket connections continue to work without issue but any new connection attempts have an extremely high latency and packet loss. I'm not comfortable sharing this packet capture though because it could potentially contain some authentication info which could be reverse engineered..
I'll paste a snippet of it below.
I'm going to see if my ISP will send me a replacement ONT so that I can cover my bases there.
1 2020/231 17:46:10.570391 0.000000000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=123 2 2020/231 17:46:10.570398 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=252 3 2020/231 17:46:10.581041 0.010643000 71.36.120.123 205.251.197.145 DNS Standard query 0x5e31 A a.teads.tv OPT 4 2020/231 17:46:10.604454 0.023413000 71.36.120.123 182.161.72.6 DNS Standard query 0x8f03 AAAA ns28.criteo.com OPT 5 2020/231 17:46:10.604504 0.000050000 71.36.120.123 74.119.118.255 DNS Standard query 0x7900 AAAA ns22.criteo.com OPT 6 2020/231 17:46:10.604535 0.000031000 71.36.120.123 74.119.118.255 DNS Standard query 0xa752 AAAA ns27.criteo.com OPT 7 2020/231 17:46:10.604866 0.000331000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=128 8 2020/231 17:46:10.610319 0.005453000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=70 9 2020/231 17:46:10.612201 0.001882000 71.36.120.123 108.162.193.135 DNS Standard query 0xcf67 A ns.wpopt.net OPT 10 2020/231 17:46:10.612759 0.000558000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=42 11 2020/231 17:46:10.616211 0.003452000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=128 12 2020/231 17:46:10.623546 0.007335000 71.36.120.123 192.112.36.4 DNS Standard query 0xbbf4 A wpad.britannia.local OPT 13 2020/231 17:46:10.644779 0.021233000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=126 14 2020/231 17:46:10.644786 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=73 15 2020/231 17:46:10.644935 0.000149000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=627 16 2020/231 17:46:10.644941 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=627 17 2020/231 17:46:10.656326 0.011385000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1139 18 2020/231 17:46:10.662098 0.005772000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=125 19 2020/231 17:46:10.662255 0.000157000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1139 20 2020/231 17:46:10.663498 0.001243000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 21 2020/231 17:46:10.667730 0.004232000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1139 22 2020/231 17:46:10.671272 0.003542000 71.36.120.123 192.112.36.4 DNS Standard query 0x06ce A local OPT 23 2020/231 17:46:10.673319 0.002047000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1139 24 2020/231 17:46:10.673900 0.000581000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 25 2020/231 17:46:10.678912 0.005012000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=119 26 2020/231 17:46:10.683979 0.005067000 71.36.120.123 192.112.36.4 DNS Standard query 0x5d0d AAAA ns-1881.awsdns-43.co.uk OPT 27 2020/231 17:46:10.684778 0.000799000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1139 28 2020/231 17:46:10.684785 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 29 2020/231 17:46:10.690549 0.005764000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 30 2020/231 17:46:10.695846 0.005297000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 31 2020/231 17:46:10.696177 0.000331000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 32 2020/231 17:46:10.701793 0.005616000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=114 33 2020/231 17:46:10.701950 0.000157000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=243 34 2020/231 17:46:10.701956 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 35 2020/231 17:46:10.707201 0.005245000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 36 2020/231 17:46:10.712448 0.005247000 71.36.120.123 192.112.36.4 DNS Standard query 0xe812 A ns-1881.awsdns-43.co.uk OPT 37 2020/231 17:46:10.713558 0.001110000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 38 2020/231 17:46:10.715311 0.001753000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=50 39 2020/231 17:46:10.718780 0.003469000 71.36.120.123 204.13.251.136 DNS Standard query 0xb24a A ns3.p29.dynect.net OPT 40 2020/231 17:46:10.718817 0.000037000 71.36.120.123 156.154.65.210 DNS Standard query 0xd119 A elb-ore-amz.nimbus.bitdefender.net OPT 41 2020/231 17:46:10.719147 0.000330000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=114 42 2020/231 17:46:10.719154 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 43 2020/231 17:46:10.724919 0.005765000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 44 2020/231 17:46:10.730536 0.005617000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 45 2020/231 17:46:10.736308 0.005772000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=114 46 2020/231 17:46:10.736315 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 47 2020/231 17:46:10.741924 0.005609000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=46 48 2020/231 17:46:10.741931 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1140 49 2020/231 17:46:10.742357 0.000426000 71.36.120.123 74.125.250.87 STUN Binding Request user: QUk4jW0q5FYFBAXl:R6ng 50 2020/231 17:46:10.755482 0.013125000 74.125.250.87 71.36.120.123 STUN Binding Success Response user: QUk4jW0q5FYFBAXl:R6ng XOR-MAPPED-ADDRESS: 71.36.120.123:7162 51 2020/231 17:46:10.759071 0.003589000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=98 52 2020/231 17:46:10.759078 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=39 53 2020/231 17:46:10.766467 0.007389000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 54 2020/231 17:46:10.770543 0.004076000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=107 55 2020/231 17:46:10.770549 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=702 56 2020/231 17:46:10.771652 0.001103000 71.36.120.123 162.88.61.21 DNS Standard query 0x78ad A ns2.p29.dynect.net OPT 57 2020/231 17:46:10.776351 0.004699000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=77 58 2020/231 17:46:10.783203 0.006852000 71.36.120.123 205.251.193.209 DNS Standard query 0x4fc2 A ns-645.awsdns-16.net OPT 59 2020/231 17:46:10.783285 0.000082000 71.36.120.123 162.88.60.21 DNS Standard query 0xb882 A ns1.p29.dynect.net OPT 60 2020/231 17:46:10.799214 0.015929000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=75 61 2020/231 17:46:10.803738 0.004524000 216.239.38.10 71.36.120.123 DNS Standard query response 0x021f A mobile-gtalk.l.google.com A 74.125.195.188 OPT 62 2020/231 17:46:10.814661 0.010923000 192.35.51.30 71.36.120.123 DNS Standard query response 0x43c6 A dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns3.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NSEC3 RRSIG AAAA 2001:500:90:1::136 A 208.78.70.136 OPT 63 2020/231 17:46:10.816580 0.001919000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=69 64 2020/231 17:46:10.817005 0.000425000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=42 65 2020/231 17:46:10.822015 0.005010000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=76 66 2020/231 17:46:10.825603 0.003588000 199.19.54.1 71.36.120.123 DNS Standard query response 0x29cb A ultradns.org OPT 67 2020/231 17:46:10.825684 0.000081000 71.36.120.123 199.19.54.1 TCP 0 0 1 0 65228 44963 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=3298201888 TSecr=0 68 2020/231 17:46:10.827733 0.002049000 199.249.120.1 71.36.120.123 DNS Standard query response 0xd56c AAAA ns3-06.azure-dns.org OPT 69 2020/231 17:46:10.827740 0.000007000 65.22.162.17 71.36.120.123 TCP 0 0 1 1 65535 53 → 44907 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1452 WS=64 SACK_PERM=1 TSval=734570156 TSecr=474470959 70 2020/231 17:46:10.827793 0.000053000 71.36.120.123 199.249.120.1 TCP 0 0 1 0 65228 44964 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=1310942176 TSecr=0 71 2020/231 17:46:10.827892 0.000099000 199.19.53.1 71.36.120.123 TCP 0 1 1 0 0 53 → 44896 [RST] Seq=1 Win=0 Len=0 72 2020/231 17:46:10.827899 0.000007000 192.48.79.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44909 [RST] Seq=1 Win=0 Len=0 73 2020/231 17:46:10.829917 0.002018000 198.51.45.66 71.36.120.123 DNS Standard query response 0x8e05 A tlx.3lift.com CNAME us-west-tlx.3lift.com CNAME dualstack.exchange-prod-582331669.us-west-1.elb.amazonaws.com OPT 74 2020/231 17:46:10.831699 0.001782000 71.36.120.123 64.4.48.3 DNS Standard query 0x1f8a A ns2-34.azure-dns.net OPT 75 2020/231 17:46:10.831732 0.000033000 71.36.120.123 205.251.199.144 DNS Standard query 0xf8f8 A ns-645.awsdns-16.net OPT 76 2020/231 17:46:10.837963 0.006231000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=35 77 2020/231 17:46:10.839231 0.001268000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=84 78 2020/231 17:46:10.841102 0.001871000 199.19.53.1 71.36.120.123 TCP 0 0 1 1 65535 53 → 44916 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1452 WS=64 SACK_PERM=1 TSval=3678409839 TSecr=721965227 79 2020/231 17:46:10.841109 0.000007000 192.5.6.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44858 [RST] Seq=1 Win=0 Len=0 80 2020/231 17:46:10.843287 0.002178000 192.36.148.17 71.36.120.123 DNS Standard query response 0x801c No such name A bidder.criteo.com.britannia.local OPT 81 2020/231 17:46:10.845318 0.002031000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=246 82 2020/231 17:46:10.845324 0.000006000 2.22.230.67 71.36.120.123 DNS Standard query response 0x7135 A a16-65.akam.net A 23.211.132.65 OPT 83 2020/231 17:46:10.845601 0.000277000 71.36.120.123 96.7.49.67 DNS Standard query 0x8b78 AAAA a16-65.akam.net OPT 84 2020/231 17:46:10.847337 0.001736000 199.7.91.13 71.36.120.123 DNS Standard query response 0x2c86 No such name A local NSEC locker RRSIG OPT 85 2020/231 17:46:10.849520 0.002183000 192.48.79.30 71.36.120.123 DNS Standard query response 0x5869 A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT 86 2020/231 17:46:10.849580 0.000060000 71.36.120.123 192.48.79.30 TCP 0 0 1 0 65228 44965 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=144028582 TSecr=0 87 2020/231 17:46:10.850848 0.001268000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 88 2020/231 17:46:10.851470 0.000622000 2.22.230.67 71.36.120.123 DNS Standard query response 0x31af A a5-64.akam.net A 95.100.168.64 OPT 89 2020/231 17:46:10.851627 0.000157000 192.48.79.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44909 [RST] Seq=1 Win=0 Len=0 90 2020/231 17:46:10.851722 0.000095000 71.36.120.123 96.7.49.67 DNS Standard query 0x9ef7 A a5-64.akam.net OPT 91 2020/231 17:46:10.853770 0.002048000 213.248.216.1 71.36.120.123 DNS Standard query response 0xb526 A ns-1881.awsdns-43.co.uk NS g-ns-363.awsdns-43.co.uk NS g-ns-939.awsdns-43.co.uk NS g-ns-1518.awsdns-43.co.uk NS g-ns-1839.awsdns-43.co.uk NSEC3 RRSIG OPT 92 2020/231 17:46:10.856579 0.002809000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 93 2020/231 17:46:10.862037 0.005458000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=87 94 2020/231 17:46:10.862193 0.000156000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 95 2020/231 17:46:10.865180 0.002987000 71.36.120.123 216.252.166.11 DNS Standard query 0x4a91 A ib.adnxs.com OPT 96 2020/231 17:46:10.866755 0.001575000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 97 2020/231 17:46:10.867868 0.001113000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 98 2020/231 17:46:10.871459 0.003591000 71.36.120.123 156.154.65.210 DNS Standard query 0x451e A elb-ore-amz.nimbus.bitdefender.net OPT 99 2020/231 17:46:10.879433 0.007974000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=90 100 2020/231 17:46:10.879440 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 101 2020/231 17:46:10.879589 0.000149000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 102 2020/231 17:46:10.890823 0.011234000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 103 2020/231 17:46:10.896595 0.005772000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=94 104 2020/231 17:46:10.902368 0.005773000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=59 105 2020/231 17:46:10.902374 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1173 106 2020/231 17:46:10.902380 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=1174 107 2020/231 17:46:10.908063 0.005683000 71.36.120.123 200.7.86.53 DNS Standard query 0x760f PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa OPT 108 2020/231 17:46:10.913699 0.005636000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=73 109 2020/231 17:46:10.917560 0.003861000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 110 2020/231 17:46:10.919297 0.001737000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=96 111 2020/231 17:46:10.919452 0.000155000 96.7.49.67 71.36.120.123 DNS Standard query response 0x351a A a16-65.akam.net A 23.211.132.65 OPT 112 2020/231 17:46:10.919458 0.000006000 81.17.242.98 71.36.120.123 ICMP Time-to-live exceeded (Time to live exceeded in transit) 113 2020/231 17:46:10.919465 0.000007000 81.17.242.98 71.36.120.123 ICMP Time-to-live exceeded (Time to live exceeded in transit) 114 2020/231 17:46:10.919717 0.000252000 71.36.120.123 23.211.133.67 DNS Standard query 0xb7b3 A a16-65.akam.net OPT 115 2020/231 17:46:10.921607 0.001890000 96.7.49.67 71.36.120.123 DNS Standard query response 0x8d3e AAAA use2.akam.net SOA internal.akam.net OPT 116 2020/231 17:46:10.921867 0.000260000 71.36.120.123 23.211.133.67 DNS Standard query 0x2bb5 AAAA use2.akam.net OPT 117 2020/231 17:46:10.923758 0.001891000 199.253.182.182 71.36.120.123 DNS Standard query response 0xfdd1 PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa NS r.arin.net NS u.arin.net NS x.arin.net NS y.arin.net NS z.arin.net NS arin.authdns.ripe.net DS RRSIG OPT 118 2020/231 17:46:10.925005 0.001247000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=697 119 2020/231 17:46:10.925786 0.000781000 199.19.56.1 71.36.120.123 DNS Standard query response 0x3f87 A ultradns.org OPT 120 2020/231 17:46:10.925846 0.000060000 71.36.120.123 199.19.56.1 TCP 0 0 1 0 65228 44966 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=923319125 TSecr=0 121 2020/231 17:46:10.927895 0.002049000 198.51.44.2 71.36.120.123 DNS Standard query response 0xb0f4 A prebid.appnexusgslb.net A 68.67.129.85 OPT 122 2020/231 17:46:10.929922 0.002027000 65.22.163.17 71.36.120.123 DNS Standard query response 0x4662 A dmx.districtm.io OPT 123 2020/231 17:46:10.930703 0.000781000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=39 124 2020/231 17:46:10.931952 0.001249000 23.211.133.67 71.36.120.123 DNS Standard query response 0x4d8f AAAA a28-67.akam.net SOA internal.akam.net OPT 125 2020/231 17:46:10.932073 0.000121000 71.36.120.123 184.85.248.67 DNS Standard query 0x2361 AAAA a28-67.akam.net OPT 126 2020/231 17:46:10.934120 0.002047000 192.36.148.17 71.36.120.123 DNS Standard query response 0x940b DNSKEY <Root> OPT 127 2020/231 17:46:10.936146 0.002026000 23.211.133.67 71.36.120.123 DNS Standard query response 0x0d65 A a5-64.akam.net A 95.100.168.64 OPT 128 2020/231 17:46:10.936393 0.000247000 71.36.120.123 95.101.36.67 DNS Standard query 0x51e0 AAAA a5-64.akam.net OPT 129 2020/231 17:46:10.942964 0.006571000 192.48.79.30 71.36.120.123 DNS Standard query response 0x4b33 A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT 130 2020/231 17:46:10.943024 0.000060000 71.36.120.123 192.48.79.30 TCP 0 0 1 0 65228 44967 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2408885028 TSecr=0 131 2020/231 17:46:10.945072 0.002048000 199.254.48.1 71.36.120.123 DNS Standard query response 0xd584 AAAA ns4-06.azure-dns.info OPT 132 2020/231 17:46:10.947099 0.002027000 199.254.48.1 71.36.120.123 DNS Standard query response 0xb8a3 A ns4-06.azure-dns.info OPT 133 2020/231 17:46:10.947881 0.000782000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=85 134 2020/231 17:46:10.949130 0.001249000 199.253.182.182 71.36.120.123 DNS Standard query response 0xf9cc PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa NS r.arin.net NS u.arin.net NS x.arin.net NS y.arin.net NS z.arin.net NS arin.authdns.ripe.net DS RRSIG OPT 135 2020/231 17:46:10.950288 0.001158000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=46 136 2020/231 17:46:10.951244 0.000956000 182.161.72.6 71.36.120.123 DNS Standard query response 0x1f80 AAAA ns26.criteo.com SOA ns23.criteo.com OPT 137 2020/231 17:46:10.951251 0.000007000 8.8.8.8 71.36.120.123 ICMP Echo (ping) reply id=0x0e8f, seq=389/34049, ttl=118 138 2020/231 17:46:10.953275 0.002024000 172.217.14.196 71.36.120.123 TCP 0 0 1 1 65535 80 → 25037 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=2310718172 TSecr=149079597 WS=256 139 2020/231 17:46:10.953430 0.000155000 192.48.79.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44909 [RST] Seq=1 Win=0 Len=0 140 2020/231 17:46:10.955459 0.002029000 96.7.49.67 71.36.120.123 DNS Standard query response 0x6fcb A as-sec.casalemedia.com CNAME as-sec.casalemedia.com.edgekey.net OPT 141 2020/231 17:46:10.959517 0.004058000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=84 142 2020/231 17:46:10.959829 0.000312000 192.48.79.30 71.36.120.123 DNS Standard query response 0x4c0b A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT 143 2020/231 17:46:10.959895 0.000066000 71.36.120.123 192.48.79.30 TCP 0 0 1 0 65228 44968 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=3466359485 TSecr=0 144 2020/231 17:46:10.961788 0.001893000 192.112.36.4 71.36.120.123 DNS Standard query response 0xcbc7 No such name A wpad.britannia.local OPT 145 2020/231 17:46:10.962221 0.000433000 71.36.120.123 192.12.94.30 DNS Standard query 0x61b2 A appnexusgslb.com OPT 146 2020/231 17:46:10.963956 0.001735000 96.7.49.67 71.36.120.123 DNS Standard query response 0x1a52 A a16-65.akam.net A 23.211.132.65 OPT 147 2020/231 17:46:10.964207 0.000251000 71.36.120.123 95.101.36.67 DNS Standard query 0xf4ef A a16-65.akam.net OPT 148 2020/231 17:46:10.967801 0.003594000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=42 149 2020/231 17:46:10.982330 0.014529000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=79 150 2020/231 17:46:10.982337 0.000007000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=327 151 2020/231 17:46:10.985450 0.003113000 172.217.14.196 71.36.120.123 TCP 0 0 1 1 65535 443 → 39665 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=2778687698 TSecr=149079006 WS=256 152 2020/231 17:46:10.985456 0.000006000 172.217.14.195 71.36.120.123 TCP 0 0 1 1 65535 80 → 6268 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=1227369459 TSecr=149079007 WS=256 153 2020/231 17:46:10.987478 0.002022000 156.154.101.3 71.36.120.123 DNS Standard query response 0x1d19 AAAA nsc.nic.uk SOA dns1.nic.uk OPT 154 2020/231 17:46:10.989118 0.001640000 71.36.120.123 173.245.59.135 DNS Standard query 0x4482 A ns.wpopt.net OPT 155 2020/231 17:46:10.989148 0.000030000 71.36.120.123 172.64.33.135 DNS Standard query 0x5a54 A ns.wpopt.net OPT 156 2020/231 17:46:10.989633 0.000485000 96.7.49.67 71.36.120.123 DNS Standard query response 0xbb37 AAAA a9-67.akam.net AAAA 2a02:26f0:117::43 OPT 157 2020/231 17:46:10.989875 0.000242000 71.36.120.123 184.85.248.67 DNS Standard query 0xbdfc AAAA a9-67.akam.net OPT 158 2020/231 17:46:10.990877 0.001002000 71.36.120.123 172.217.14.196 TCP 0 1 1 1 343 25037 → 80 [ACK] Seq=1 Ack=1 Win=343 Len=0 TSval=149080659 TSecr=2310718172 159 2020/231 17:46:10.990892 0.000015000 71.36.120.123 172.217.14.196 TCP 0 882 882 1076 354 [TCP ACKed unseen segment] 39665 → 443 [ACK] Seq=882 Ack=1076 Win=354 Len=0 TSval=149080659 TSecr=2778687985 SLE=0 SRE=1 160 2020/231 17:46:10.991098 0.000206000 71.36.120.123 172.217.14.195 TCP 0 229 229 103 343 [TCP ACKed unseen segment] 6268 → 80 [ACK] Seq=229 Ack=103 Win=343 Len=0 TSval=149080659 TSecr=1227369723 SLE=0 SRE=1 161 2020/231 17:46:10.991362 0.000264000 71.36.120.123 172.217.14.196 HTTP 207 1 208 1 343 207 GET /gen_204 HTTP/1.1 162 2020/231 17:46:10.993567 0.002205000 176.32.99.148 71.36.120.123 TLSv1.2 46 1 47 1 2188 46 Application Data 163 2020/231 17:46:10.996531 0.002964000 172.217.14.196 71.36.120.123 TCP 0 1 1 208 66816 80 → 25037 [ACK] Seq=1 Ack=208 Win=66816 Len=0 TSval=2310719898 TSecr=149080659 164 2020/231 17:46:10.999340 0.002809000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=79 165 2020/231 17:46:11.000038 0.000698000 71.36.120.123 176.32.99.148 TLSv1.2 46 1 47 47 8209 46 Application Data 166 2020/231 17:46:11.002999 0.002961000 71.36.120.123 198.51.45.2 DNS Standard query 0x3b38 A tlx.3lift.com OPT 167 2020/231 17:46:11.005044 0.002045000 192.48.79.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44909 [RST] Seq=1 Win=0 Len=0 168 2020/231 17:46:11.007073 0.002029000 192.112.36.4 71.36.120.123 DNS Standard query response 0xd94a No such name A local OPT 169 2020/231 17:46:11.008634 0.001561000 172.217.14.196 71.36.120.123 HTTP 314 1 315 208 66816 314 HTTP/1.1 204 No Content 170 2020/231 17:46:11.009259 0.000625000 192.112.36.4 71.36.120.123 DNS Standard query response 0xcd80 AAAA ns-1881.awsdns-43.co.uk OPT 171 2020/231 17:46:11.010640 0.001381000 71.36.120.123 172.217.14.196 TCP 0 208 208 315 347 25037 → 80 [ACK] Seq=208 Ack=315 Win=347 Len=0 TSval=149080664 TSecr=2310719910 172 2020/231 17:46:11.011283 0.000643000 23.211.133.67 71.36.120.123 DNS Standard query response 0x8337 A a16-65.akam.net A 23.211.132.65 OPT 173 2020/231 17:46:11.011532 0.000249000 71.36.120.123 95.100.173.67 DNS Standard query 0x5f5c AAAA a16-65.akam.net OPT 174 2020/231 17:46:11.013225 0.001693000 71.36.120.123 172.217.14.196 TCP 0 208 209 315 347 25037 → 80 [FIN, ACK] Seq=208 Ack=315 Win=347 Len=0 TSval=149080665 TSecr=2310719910 175 2020/231 17:46:11.013400 0.000175000 2.22.230.67 71.36.120.123 DNS Standard query response 0x0068 AAAA a22-67.akam.net SOA internal.akam.net OPT 176 2020/231 17:46:11.013406 0.000006000 81.17.242.98 71.36.120.123 ICMP Time-to-live exceeded (Time to live exceeded in transit) 177 2020/231 17:46:11.013535 0.000129000 71.36.120.123 95.100.173.67 DNS Standard query 0x2cbf AAAA a22-67.akam.net OPT 178 2020/231 17:46:11.015427 0.001892000 192.112.36.4 71.36.120.123 DNS Standard query response 0x5271 A ns-1881.awsdns-43.co.uk OPT 179 2020/231 17:46:11.018265 0.002838000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=42 180 2020/231 17:46:11.019300 0.001035000 71.36.120.123 8.8.8.8 ICMP Echo (ping) request id=0x0e8f, seq=396/35841, ttl=64 (no response found!) 181 2020/231 17:46:11.019344 0.000044000 71.36.120.123 192.12.94.30 DNS Standard query 0xf06c A ns27.domaincontrol.com OPT 182 2020/231 17:46:11.019378 0.000034000 71.36.120.123 192.12.94.30 DNS Standard query 0x0b0c AAAA ns27.domaincontrol.com OPT 183 2020/231 17:46:11.019410 0.000032000 71.36.120.123 192.12.94.30 DNS Standard query 0xaaef AAAA ns28.domaincontrol.com OPT 184 2020/231 17:46:11.022237 0.002827000 156.154.65.210 71.36.120.123 DNS Standard query response 0x8827 A elb-ore-amz.nimbus.bitdefender.net CNAME kube-nimbus-471965604.us-west-2.elb.amazonaws.com OPT 185 2020/231 17:46:11.022391 0.000154000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=80 186 2020/231 17:46:11.024265 0.001874000 204.13.251.136 71.36.120.123 DNS Standard query response 0x4bf1 A ns3.p29.dynect.net A 208.78.71.29 NS ns3.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net OPT 187 2020/231 17:46:11.026294 0.002029000 162.88.61.21 71.36.120.123 DNS Standard query response 0x4046 A ns2.p29.dynect.net A 204.13.250.29 OPT 188 2020/231 17:46:11.026762 0.000468000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=66 189 2020/231 17:46:11.028484 0.001722000 192.58.128.30 71.36.120.123 DNS Standard query response 0x47b3 A biz NS k.gtld.biz NS f.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz DS DS RRSIG OPT 190 2020/231 17:46:11.030513 0.002029000 192.5.5.241 71.36.120.123 DNS Standard query response 0x724c A biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT 191 2020/231 17:46:11.030578 0.000065000 71.36.120.123 192.5.5.241 TCP 0 0 1 0 65228 44969 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2411125480 TSecr=0 192 2020/231 17:46:11.032627 0.002049000 192.5.5.241 71.36.120.123 DNS Standard query response 0x847d A e.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT 193 2020/231 17:46:11.032683 0.000056000 71.36.120.123 192.5.5.241 TCP 0 0 1 0 65228 44970 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=259106889 TSecr=0 194 2020/231 17:46:11.036916 0.004233000 204.13.250.136 71.36.120.123 DNS Standard query response 0xfdbf A ns2.p29.dynect.net A 204.13.250.29 NS ns3.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net OPT 195 2020/231 17:46:11.037107 0.000191000 71.36.120.123 208.78.71.136 DNS Standard query 0x9ddf A ns2.p29.dynect.net OPT 196 2020/231 17:46:11.038999 0.001892000 205.251.195.18 71.36.120.123 DNS Standard query response 0x4d9e A ns-645.awsdns-16.net A 205.251.194.133 NS g-ns-1360.awsdns-16.net NS g-ns-1936.awsdns-16.net NS g-ns-465.awsdns-16.net NS g-ns-786.awsdns-16.net A 205.251.197.80 AAAA 2600:9000:5305:5000::1 A 205.251.199.144 AAAA 2600:9000:5307:9000::1 A 205.251.193.209 AAAA 2600:9000:5301:d100::1 A 205.251.195.18 AAAA 2600:9000:5303:1200::1 OPT 197 2020/231 17:46:11.039007 0.000008000 192.35.51.30 71.36.120.123 TCP 0 1 1 0 0 53 → 44915 [RST] Seq=1 Win=0 Len=0 198 2020/231 17:46:11.039467 0.000460000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=83 199 2020/231 17:46:11.039473 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=620 200 2020/231 17:46:11.041027 0.001554000 192.42.93.30 71.36.120.123 DNS Standard query response 0x4e36 A amplitude.com NS ns-579.awsdns-08.net NS ns-260.awsdns-32.com NS ns-1262.awsdns-29.org NS ns-1942.awsdns-50.co.uk NSEC3 RRSIG A 205.251.193.4 OPT 201 2020/231 17:46:11.043056 0.002029000 198.97.190.53 71.36.120.123 DNS Standard query response 0x47e6 A biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT 202 2020/231 17:46:11.043114 0.000058000 71.36.120.123 198.97.190.53 TCP 0 0 1 0 65228 44971 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2608320456 TSecr=0 203 2020/231 17:46:11.045161 0.002047000 162.88.60.21 71.36.120.123 DNS Standard query response 0x346c A ns1.p29.dynect.net A 208.78.70.29 OPT 204 2020/231 17:46:11.047467 0.002306000 71.36.120.123 216.239.34.10 DNS Standard query 0xa45d A mobile-gtalk.l.google.com OPT 205 2020/231 17:46:11.050920 0.003453000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=79 206 2020/231 17:46:11.050926 0.000006000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=107 207 2020/231 17:46:11.053726 0.002800000 205.251.193.209 71.36.120.123 DNS Standard query response 0x79d3 A ns-645.awsdns-16.net A 205.251.194.133 NS g-ns-1360.awsdns-16.net NS g-ns-1936.awsdns-16.net NS g-ns-465.awsdns-16.net NS g-ns-786.awsdns-16.net A 205.251.197.80 AAAA 2600:9000:5305:5000::1 A 205.251.199.144 AAAA 2600:9000:5307:9000::1 A 205.251.193.209 AAAA 2600:9000:5301:d100::1 A 205.251.195.18 AAAA 2600:9000:5303:1200::1 OPT 208 2020/231 17:46:11.055755 0.002029000 156.154.65.210 71.36.120.123 DNS Standard query response 0x0f00 A elb-ore-amz.nimbus.bitdefender.net CNAME kube-nimbus-471965604.us-west-2.elb.amazonaws.com OPT 209 2020/231 17:46:11.057944 0.002189000 64.4.48.1 71.36.120.123 DNS Standard query response 0x3e3f A ns2-34.azure-dns.net A 150.171.16.34 OPT 210 2020/231 17:46:11.059971 0.002027000 205.251.194.68 71.36.120.123 DNS Standard query response 0x3039 AAAA ns-38.awsdns-04.com AAAA 2600:9000:5300:2600::1 NS g-ns-1156.awsdns-04.com NS g-ns-1732.awsdns-04.com NS g-ns-5.awsdns-04.com NS g-ns-580.awsdns-04.com A 205.251.196.132 AAAA 2600:9000:5304:8400::1 A 205.251.198.196 AAAA 2600:9000:5306:c400::1 A 205.251.192.5 AAAA 2600:9000:5300:500::1 A 205.251.194.68 AAAA 2600:9000:5302:4400::1 OPT 211 2020/231 17:46:11.062155 0.002184000 2.22.230.67 71.36.120.123 DNS Standard query response 0x6d3c A a9-67.akam.net A 184.85.248.67 OPT 212 2020/231 17:46:11.062411 0.000256000 71.36.120.123 95.100.173.67 DNS Standard query 0xd1e4 AAAA a9-67.akam.net OPT 213 2020/231 17:46:11.064145 0.001734000 43.230.48.1 71.36.120.123 DNS Standard query response 0xa2b0 AAAA nsd.nic.uk SOA dns1.nic.uk OPT 214 2020/231 17:46:11.066017 0.001872000 74.125.250.87 71.36.120.123 UDP 19305 → 7162 Len=66 215 2020/231 17:46:11.066176 0.000159000 198.97.190.53 71.36.120.123 DNS Standard query response 0x09cf A a.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT 216 2020/231 17:46:11.067344 0.001168000 71.36.120.123 74.125.250.87 UDP 7162 → 19305 Len=42 217 2020/231 17:46:11.068301 0.000957000 216.252.166.10 71.36.120.123 DNS Standard query response 0xd535 A ib.adnxs.com CNAME g.geogslb.com NS ns1.gslb.com NS ns2.gslb.com 218 2020/231 17:46:11.068307 0.000006000 81.17.242.98 71.36.120.123 ICMP Time-to-live exceeded (Time to live exceeded in transit) -
Hmm, that sure starts to looks like an upstream routing issue.
What is sending that ICMP TTL exceeded response? What was the target?
Steve
-
@stephenw10 target was the google DNS server, 8.8.8.8
So far this week, the issue has manifested like clockwork almost every day between 10:50 and 11:15 AM. with one occasion where it also reoccurred near noon as well.
I've contacted my ISP and they beleive they saw some up line issues and have a tech coming out next week...
I'm getting very tiered of this issue very fast.
-
ISP replaced the ONT and I had been problem free until today when the behaviour appeared again..
I tried to do a tracert and every hop diddnt response and the last 8.8.8.8 had a response time of 1248ms
I was able to restore my connection by going to status>interfaces and then disconnecting and recconecting the WAN PPOE.
Could use some guidance on troubleshooting PPOE issues as well as reccomendations on a scripted workaround to automatically restart it if non responsive after a period of time.
-
You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.
Steve
I'll have to grab that info the next time this behavior occurs, which specific info would I want to grab in this case?
Not sure on that specific AP, it was likely picking up traffic from some random device on my network.
Here's my config related to my PPOE wan if that helps any.
<wan> <if>pppoe0</if> <blockbogons></blockbogons> <descr><![CDATA[WAN01_CenturyLink]]></descr> <alias-address></alias-address> <alias-subnet>32</alias-subnet> <spoofmac></spoofmac> <blockpriv></blockpriv> <enable></enable> <ipaddr>pppoe</ipaddr> </wan> <vlan> <if>igb0</if> <tag>201</tag> <pcp></pcp> <descr><![CDATA[WAN_01_VLAN201]]></descr> <vlanif>igb0.201</vlanif> </vlan> <ppps> <ppp> <ptpid>0</ptpid> <type>pppoe</type> <if>pppoe0</if> <ports>igb0.201</ports> <username><![CDATA[REDACTED@centurylink.net]]></username> <password><![CDATA[REDACTED]]></password> <bandwidth></bandwidth> <mtu></mtu> <mru></mru> <mrru></mrru> </ppp> </ppps> <gateways> <gateway_item> <interface>wan</interface> <gateway>dynamic</gateway> <name>WAN01_CENTURYLINK_PPPOE</name> <weight>1</weight> <ipprotocol>inet</ipprotocol> <descr><![CDATA[Interface WAN01_CENTURYLINK_PPPOE Gateway]]></descr> <monitor>8.8.8.8</monitor> </gateway_item> <defaultgw4>WAN01_CENTURYLINK_PPPOE</defaultgw4> <defaultgw6>-</defaultgw6> </gateways> -
Nothing unusual there.
You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Nothing unusual there.
You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.
Steve
It's unfortunately sometimes occurs more frequently then that. Last event was yesterday around ~1pm and it reoccured a short bit ago around 9:20am today.
I was not able to get the connection back this time by disconnecting and reconnecting the PPOE cconnection, ended up restarting PFsense.
Next step will likely be for me to disable snort for atleast a week or until the issue returns to see if the behaviour reappears.
I'm kind of grasping at straws right now though.....
------------ System logs from time period ---------
Aug 31 09:10:20 snort 67712 [1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.77.227:33798 -> 71.36.122.177:443 Aug 31 09:10:57 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 183.131.3.210:58864 -> 71.36.122.177:1433 Aug 31 09:11:25 snort 67712 [1:2403368:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 35 [Classification: Misc Attack] [Priority: 2] {TCP} 51.161.12.231:32767 -> 71.36.122.177:8545 Aug 31 09:13:13 snort 67712 [1:2403448:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 75 [Classification: Misc Attack] [Priority: 2] {TCP} 89.248.168.157:37856 -> 71.36.122.177:41065 Aug 31 09:14:38 snort 67712 [1:2403458:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 80 [Classification: Misc Attack] [Priority: 2] {TCP} 92.63.197.55:40327 -> 71.36.122.177:3377 Aug 31 09:15:07 snort 67712 [1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.56.238:55872 -> 71.36.122.177:5900 Aug 31 09:16:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.20:57576 -> 71.36.122.177:3345 Aug 31 09:16:14 rc.gateway_alarm 27046 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:506.622ms RTTsd:787.570ms Loss:0%) Aug 31 09:16:14 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:16:14 check_reload_status Restarting ipsec tunnels Aug 31 09:16:14 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:16:14 check_reload_status Reloading filter Aug 31 09:16:15 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:16:15 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:07 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.3:55957 -> 71.36.122.177:3310 Aug 31 09:17:07 snort 67712 [1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291 Aug 31 09:17:07 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291 Aug 31 09:17:22 rc.gateway_alarm 11126 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4120.023ms RTTsd:1799.455ms Loss:22%) Aug 31 09:17:22 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:17:22 check_reload_status Restarting ipsec tunnels Aug 31 09:17:22 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:17:22 check_reload_status Reloading filter Aug 31 09:17:23 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:17:23 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:27 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852 Aug 31 09:17:27 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852 Aug 31 09:17:35 rc.gateway_alarm 61503 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3703.111ms RTTsd:2201.113ms Loss:11%) Aug 31 09:17:35 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:17:35 check_reload_status Restarting ipsec tunnels Aug 31 09:17:35 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:17:35 check_reload_status Reloading filter Aug 31 09:17:36 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:17:36 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:17:38 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.102:47924 -> 71.36.122.177:26098 Aug 31 09:18:31 snort 67712 [1:2403424:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 63 [Classification: Misc Attack] [Priority: 2] {TCP} 78.108.177.54:26525 -> 71.36.122.177:8080 Aug 31 09:18:32 rc.gateway_alarm 50465 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:0 RTT:310.577ms RTTsd:435.870ms Loss:0%) Aug 31 09:18:32 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:18:32 check_reload_status Restarting ipsec tunnels Aug 31 09:18:32 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:18:32 check_reload_status Reloading filter Aug 31 09:18:33 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:18:34 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:18:57 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.91:45181 -> 71.36.122.177:33355 Aug 31 09:19:52 snort 67712 [1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.113:42826 -> 71.36.122.177:3391 Aug 31 09:20:03 snort 67712 [1:2400005:2773] ET DROP Spamhaus DROP Listed Traffic Inbound group 6 [Classification: Misc Attack] [Priority: 2] {TCP} 103.215.80.70:6000 -> 71.36.122.177:6780 Aug 31 09:20:44 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573 Aug 31 09:20:44 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573 Aug 31 09:22:03 snort 67712 [1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060 Aug 31 09:22:03 snort 67712 [1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060 Aug 31 09:22:27 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433 Aug 31 09:22:29 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433 Aug 31 09:24:01 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.4:55935 -> 71.36.122.177:835 Aug 31 09:24:26 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.8:55838 -> 71.36.122.177:4004 Aug 31 09:26:21 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124 Aug 31 09:26:21 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124 Aug 31 09:27:05 snort 67712 [1:2403406:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 54 [Classification: Misc Attack] [Priority: 2] {TCP} 62.171.161.187:43973 -> 71.36.122.177:81 Aug 31 09:28:11 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606 Aug 31 09:28:11 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606 Aug 31 09:28:47 snort 67712 [1:2403429:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 65 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.212:48824 -> 71.36.122.177:49154 Aug 31 09:28:52 rc.gateway_alarm 69361 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:502.168ms RTTsd:986.015ms Loss:0%) Aug 31 09:28:52 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:28:52 check_reload_status Restarting ipsec tunnels Aug 31 09:28:52 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:28:52 check_reload_status Reloading filter Aug 31 09:28:53 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:28:53 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:28:56 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.53:57620 -> 71.36.122.177:6357 Aug 31 09:29:02 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139 Aug 31 09:29:02 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139 Aug 31 09:29:12 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856 Aug 31 09:29:12 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856 Aug 31 09:29:44 snort 67712 [1:2403419:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 60 [Classification: Misc Attack] [Priority: 2] {UDP} 71.6.158.166:32064 -> 71.36.122.177:389 Aug 31 09:30:04 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867 Aug 31 09:30:04 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867 Aug 31 09:30:14 snort 67712 [1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060 Aug 31 09:30:14 snort 67712 [1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060 Aug 31 09:30:26 snort 67712 [1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.11:48084 -> 71.36.122.177:10552 Aug 31 09:31:13 rc.gateway_alarm 93277 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4050.647ms RTTsd:1954.397ms Loss:21%) Aug 31 09:31:13 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:31:13 check_reload_status Restarting ipsec tunnels Aug 31 09:31:13 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:31:13 check_reload_status Reloading filter Aug 31 09:31:14 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:31:14 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:31:23 rc.gateway_alarm 78618 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4322.346ms RTTsd:1981.268ms Loss:14%) Aug 31 09:31:23 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:31:23 check_reload_status Restarting ipsec tunnels Aug 31 09:31:23 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:31:23 check_reload_status Reloading filter Aug 31 09:31:24 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:31:24 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:32:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.174:44528 -> 71.36.122.177:33339 Aug 31 09:32:41 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872 Aug 31 09:32:41 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872 Aug 31 09:32:58 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015 Aug 31 09:32:58 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015 Aug 31 09:33:17 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.94:45253 -> 71.36.122.177:33384 Aug 31 09:33:56 snort 67712 [1:2403431:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 66 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.245:44258 -> 71.36.122.177:120 Aug 31 09:34:18 snort 67712 [1:2403436:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 69 [Classification: Misc Attack] [Priority: 2] {TCP} 83.97.20.35:48991 -> 71.36.122.177:6664 Aug 31 09:34:28 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.21:56468 -> 71.36.122.177:22979 Aug 31 09:35:11 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.230:40882 -> 71.36.122.177:3997 Aug 31 09:35:15 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.14:49426 -> 71.36.122.177:26187 Aug 31 09:35:25 snort 67712 [1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.60:53196 -> 71.36.122.177:4184 Aug 31 09:35:38 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139 Aug 31 09:35:38 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139 Aug 31 09:36:18 snort 67712 [1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 124.114.177.237:10566 -> 71.36.122.177:1433 Aug 31 09:36:35 snort 67712 [1:2403492:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 97 [Classification: Misc Attack] [Priority: 2] {TCP} 106.13.48.122:57394 -> 71.36.122.177:774 Aug 31 09:36:39 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548 Aug 31 09:36:39 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548 Aug 31 09:36:59 snort 67712 [1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.65.74:58855 -> 71.36.122.177:6000 Aug 31 09:37:09 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956 Aug 31 09:37:09 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956 Aug 31 09:37:11 snort 67712 [1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.22:56634 -> 71.36.122.177:33046 Aug 31 09:37:31 snort 67712 [1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547 Aug 31 09:37:31 snort 67712 [1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547 Aug 31 09:37:33 rc.gateway_alarm 53811 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4054.569ms RTTsd:2049.170ms Loss:21%) Aug 31 09:37:33 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 09:37:33 check_reload_status Restarting ipsec tunnels Aug 31 09:37:33 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 09:37:33 check_reload_status Reloading filter Aug 31 09:37:34 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 09:37:34 php-fpm /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 09:37:48 snort 67712 [1:2403372:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 37 [Classification: Misc Attack] [Priority: 2] {TCP} 54.36.109.237:50023 -> 71.36.122.177:8443---------- Gateway logs from time period ------------------
Aug 30 13:32:43 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 290791us stddev 369179us loss 0% Aug 31 09:16:14 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 506622us stddev 787570us loss 0% Aug 31 09:17:22 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4120023us stddev 1799455us loss 22% Aug 31 09:17:35 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 3703111us stddev 2201113us loss 11% Aug 31 09:18:32 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 310577us stddev 435870us loss 0% Aug 31 09:28:52 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 502168us stddev 986015us loss 0% Aug 31 09:31:13 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4050647us stddev 1954397us loss 21% Aug 31 09:31:23 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4322346us stddev 1981268us loss 14% Aug 31 09:37:33 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4054569us stddev 2049170us loss 21% Aug 31 09:40:13 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 97.120.6.183 identifier "WAN01_CENTURYLINK_PPPOE " Aug 31 09:40:30 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 507360us stddev 451625us loss 0% Aug 31 09:40:36 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 599186us stddev 671081us loss 22% Aug 31 09:40:46 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1544978us stddev 1669473us loss 11% Aug 31 09:41:13 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1609645us stddev 1562133us loss 21% Aug 31 09:41:18 dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 75.164.130.187 identifier "WAN01_CENTURYLINK_PPPOE " Aug 31 09:41:30 dpinger WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 589734us stddev 844410us loss 14%--- End logs----
I'll need to look closer at the PPP logs the next time this occurs, They were unfortunately flooded out when I restarted pfsense.
I've also been collecting data into Splunk, I'll need to go through that and set up filters when I have time today. -
Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?
-
@stephenw10 said in WAN interface stops working every few days.:
Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?
And it just occurred AGAIN, approx 2 hours later.
Restarted the router another time, This is getting very old and frustrating very fast.
I would love any guidance I can get on next steps.Bullet Points I can think of
- This behavior began a week or so after I switched from A Dell Optiplex 7010 SFF to a Poweredge R210
- Restarting PfSense or the ONT resolve the events when they occur.
- ISP has since replaced ONT.
- Config was imported from the 7010, omitting any package config.
- Have tried 3 different Nics for the Wan IF
- LAN IF is using the onboard Broadcom Nic
- Am not positive on the exact version of PFSense that was on the 7010, I had selected the stable branch and was using whatever it said was up to date.
Could there perhaps be something config related that got corrupted on import and is causing the issues?
------------- TraceRt from router WAN IF -------------------
1 * * * 2 ptld-agw1.inet.qwest.net (207.225.86.145) 1878.017 ms * * 3 * * * 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 1454.335 ms 260.238 ms 249.101 ms 5 74.125.243.177 (74.125.243.177) 158.250 ms 342.457 ms 108.170.245.113 (108.170.245.113) 1406.735 ms 6 * * * 7 * * dns.google (8.8.8.8) 1637.087 ms------------- Ping from router Wan IF ------------------------
PING 8.8.8.8 (8.8.8.8) from 71.36.127.88: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=158.006 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=544.022 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1948.327 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 158.006/883.452/1948.327/769.295 ms------------- TraceRt from router Client IF -------------------
1 ptld-dsl-gw51.ptld.qwest.net (207.225.84.51) 49.551 ms 356.669 ms 1215.833 ms 2 ptld-agw1.inet.qwest.net (207.225.86.145) 443.809 ms 1596.672 ms 1844.559 ms 3 * sea-edge-12.inet.qwest.net (67.14.41.58) 1581.644 ms 14.294 ms 4 63-158-222-114.dia.static.qwest.net (63.158.222.114) 22.815 ms 8.851 ms 8.167 ms 5 74.125.243.177 (74.125.243.177) 14.913 ms 108.170.245.97 (108.170.245.97) 8.941 ms 74.125.243.193 (74.125.243.193) 26.185 ms 6 74.125.253.67 (74.125.253.67) 169.668 ms 108.170.233.153 (108.170.233.153) 1183.524 ms 209.85.254.247 (209.85.254.247) 1935.290 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *------------- Ping from router Client IF -----------------------
PING 8.8.8.8 (8.8.8.8) from 192.168.3.1: 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=1845.914 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=2216.709 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=3239.383 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1845.914/2434.002/3239.383/589.266 ms----------------- Info from Status > Gateways -------------------
WAN01_CENTURYLINK_PPPOE (default) 207.225.84.51 8.8.4.4 1210.212ms 799.825ms 0.0% Offline Interface WAN01_CENTURYLINK_PPPOE Gateway-------------------- System Logs ---------------------------
(I tried disconnecting and reconnecting around 11:18 at which point it begins to throw Unexpected Protocol IP, Could this hint towards the issue?)Aug 31 09:58:06 check_reload_status Syncing firewall Aug 31 11:03:33 rc.gateway_alarm 87218 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:534.974ms RTTsd:880.397ms Loss:1%) Aug 31 11:03:33 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:03:33 check_reload_status Restarting ipsec tunnels Aug 31 11:03:33 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:03:33 check_reload_status Reloading filter Aug 31 11:03:34 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:03:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:05:53 rc.gateway_alarm 59267 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4196.251ms RTTsd:1499.645ms Loss:21%) Aug 31 11:05:53 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:05:53 check_reload_status Restarting ipsec tunnels Aug 31 11:05:53 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:05:53 check_reload_status Reloading filter Aug 31 11:05:54 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:05:55 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:07:44 php-fpm 73087 /index.php: Successful login for user 'admin' from: 192.168.3.157 (Local Database) Aug 31 11:07:45 rc.gateway_alarm 33853 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3838.708ms RTTsd:1985.755ms Loss:11%) Aug 31 11:07:45 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:07:45 check_reload_status Restarting ipsec tunnels Aug 31 11:07:45 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:07:45 check_reload_status Reloading filter Aug 31 11:07:46 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:07:46 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:19 rc.gateway_alarm 69490 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3395.401ms RTTsd:1821.221ms Loss:21%) Aug 31 11:10:19 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:19 check_reload_status Restarting ipsec tunnels Aug 31 11:10:19 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:19 check_reload_status Reloading filter Aug 31 11:10:20 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:20 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:29 rc.gateway_alarm 20292 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4367.359ms RTTsd:1701.643ms Loss:18%) Aug 31 11:10:29 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:29 check_reload_status Restarting ipsec tunnels Aug 31 11:10:29 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:29 check_reload_status Reloading filter Aug 31 11:10:30 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:31 php-fpm 346 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:10:32 rc.gateway_alarm 72163 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4591.740ms RTTsd:1589.594ms Loss:21%) Aug 31 11:10:32 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:10:32 check_reload_status Restarting ipsec tunnels Aug 31 11:10:32 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:10:32 check_reload_status Reloading filter Aug 31 11:10:33 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:10:34 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:11:01 rc.gateway_alarm 74351 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4430.263ms RTTsd:2115.223ms Loss:16%) Aug 31 11:11:01 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:11:01 check_reload_status Restarting ipsec tunnels Aug 31 11:11:01 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:11:01 check_reload_status Reloading filter Aug 31 11:11:02 php-fpm 346 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:11:02 php-fpm 73087 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:08 ppp caught fatal signal TERM Aug 31 11:18:08 ppp [wan] IFACE: Close event Aug 31 11:18:08 ppp [wan] IPCP: Close event Aug 31 11:18:08 ppp [wan] IPCP: state change Opened --> Closing Aug 31 11:18:08 ppp [wan] IPCP: SendTerminateReq #4 Aug 31 11:18:08 ppp [wan] IPCP: LayerDown Aug 31 11:18:08 check_reload_status Rewriting resolv.conf Aug 31 11:18:08 ppp [wan] IFACE: Down event Aug 31 11:18:08 ppp [wan] IFACE: Rename interface pppoe0 to pppoe0 Aug 31 11:18:08 ppp [wan] IPV6CP: Close event Aug 31 11:18:08 ppp [wan] IPV6CP: state change Stopped --> Closed Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:08 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:09 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan] IPCP: SendTerminateReq #5 Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP Aug 31 11:18:10 ppp [wan_link0] rec'd unexpected protocol IP **{{{{{{{{{{I deleted 60 or so more repeats of the unexpected Protocol IP error due to character limits in post.}}}}}}}}}}}}}}}** Aug 31 11:18:10 ppp [wan] Bundle: Shutdown Aug 31 11:18:10 ppp [wan_link0] Link: Shutdown Aug 31 11:18:10 ppp process 26141 terminated Aug 31 11:18:13 ppp Multi-link PPP daemon for FreeBSD Aug 31 11:18:13 ppp process 9794 started, version 5.8 (root@pfSense_v2_4_5_amd64-pfSense_v2_4_5-job-04 20:28 17-Dec-2019) Aug 31 11:18:13 ppp web: web is not running Aug 31 11:18:13 ppp [wan] Bundle: Interface ng0 created Aug 31 11:18:13 ppp [wan_link0] Link: OPEN event Aug 31 11:18:13 kernel ng0: changing name to 'pppoe0' Aug 31 11:18:13 ppp [wan_link0] LCP: Open event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan_link0] LCP: LayerStart Aug 31 11:18:13 ppp [wan_link0] PPPoE: Connecting to '' Aug 31 11:18:13 ppp PPPoE: rec'd ACNAME "ptld-dsl-gw51.ptld.qwest.net" Aug 31 11:18:13 ppp [wan_link0] PPPoE: connection successful Aug 31 11:18:13 ppp [wan_link0] Link: UP event Aug 31 11:18:13 ppp [wan_link0] LCP: Up event Aug 31 11:18:13 ppp [wan_link0] LCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Request #9 (Req-Sent) Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: SendConfigAck #9 Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] AUTHPROTO CHAP MD5 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x08202657 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Configure Ack #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan_link0] PROTOCOMP Aug 31 11:18:13 ppp [wan_link0] MRU 1492 Aug 31 11:18:13 ppp [wan_link0] MAGICNUM 0x2004df36 Aug 31 11:18:13 ppp [wan_link0] LCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan_link0] LCP: auth: peer wants CHAP, I want nothing Aug 31 11:18:13 ppp [wan_link0] LCP: LayerUp Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd CHALLENGE #244 len: 59 Aug 31 11:18:13 ppp [wan_link0] Name: "JUNOS" Aug 31 11:18:13 ppp [wan_link0] CHAP: Using authname "myerswilliam488@centurylink.net" Aug 31 11:18:13 ppp [wan_link0] CHAP: sending RESPONSE #244 len: 52 Aug 31 11:18:13 ppp [wan_link0] CHAP: rec'd SUCCESS #244 len: 4 Aug 31 11:18:13 ppp [wan_link0] LCP: authorization successful Aug 31 11:18:13 ppp [wan_link0] Link: Matched action 'bundle "wan" ""' Aug 31 11:18:13 ppp [wan_link0] Link: Join bundle "wan" Aug 31 11:18:13 ppp [wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps Aug 31 11:18:13 ppp [wan] IPCP: Open event Aug 31 11:18:13 ppp [wan] IPCP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPCP: LayerStart Aug 31 11:18:13 ppp [wan] IPV6CP: Open event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Initial --> Starting Aug 31 11:18:13 ppp [wan] IPV6CP: LayerStart Aug 31 11:18:13 ppp [wan] IPCP: Up event Aug 31 11:18:13 ppp [wan] IPCP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPV6CP: Up event Aug 31 11:18:13 ppp [wan] IPV6CP: state change Starting --> Req-Sent Aug 31 11:18:13 ppp [wan] IPV6CP: SendConfigReq #1 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Request #248 (Req-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] 207.225.84.51 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigAck #248 Aug 31 11:18:13 ppp [wan] IPADDR 207.225.84.51 Aug 31 11:18:13 ppp [wan] IPCP: state change Req-Sent --> Ack-Sent Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Reject #1 (Ack-Sent) Aug 31 11:18:13 ppp [wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #2 Aug 31 11:18:13 ppp [wan] IPADDR 0.0.0.0 Aug 31 11:18:13 ppp [wan_link0] LCP: rec'd Protocol Reject #10 (Opened) Aug 31 11:18:13 ppp [wan_link0] LCP: protocol IPV6CP was rejected Aug 31 11:18:13 ppp [wan] IPV6CP: protocol was rejected by peer Aug 31 11:18:13 ppp [wan] IPV6CP: state change Req-Sent --> Stopped Aug 31 11:18:13 ppp [wan] IPV6CP: LayerFinish Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Nak #2 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] 71.36.127.88 is OK Aug 31 11:18:13 ppp [wan] IPCP: SendConfigReq #3 Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: rec'd Configure Ack #3 (Ack-Sent) Aug 31 11:18:13 ppp [wan] IPADDR 71.36.127.88 Aug 31 11:18:13 ppp [wan] IPCP: state change Ack-Sent --> Opened Aug 31 11:18:13 ppp [wan] IPCP: LayerUp Aug 31 11:18:13 ppp [wan] 71.36.127.88 -> 207.225.84.51 Aug 31 11:18:14 check_reload_status rc.newwanip starting pppoe0 Aug 31 11:18:14 ppp [wan] IFACE: Up event Aug 31 11:18:14 ppp [wan] IFACE: Rename interface ng0 to pppoe0 Aug 31 11:18:14 rc.gateway_alarm 11603 >>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4764.745ms RTTsd:1320.248ms Loss:21%) Aug 31 11:18:14 check_reload_status updating dyndns WAN01_CENTURYLINK_PPPOE Aug 31 11:18:14 check_reload_status Restarting ipsec tunnels Aug 31 11:18:14 check_reload_status Restarting OpenVPN tunnels/interfaces Aug 31 11:18:14 check_reload_status Reloading filter Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: Info: starting on pppoe0. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: rc.newwanip: on (IP address: 71.36.127.88) (interface: WAN01_CENTURYLINK[wan]) (real interface: pppoe0). Aug 31 11:18:15 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 207.225.84.51 Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: Default gateway setting Interface WAN01_CENTURYLINK_PPPOE Gateway as default. Aug 31 11:18:15 php-fpm 73087 /rc.newwanip: IP Address has changed, killing states on former IP Address 71.36.112.131. Aug 31 11:18:16 php-fpm 347 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE. Aug 31 11:18:17 dhcpleases /etc/hosts changed size from original! Aug 31 11:18:17 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process. Aug 31 11:18:20 dhcpleases kqueue error: unknown Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wancustom''0.cache: 71.36.127.88 Aug 31 11:18:22 php-fpm 346 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 31 11:18:22 php-fpm 73087 /rc.newwanip: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Aug 31 11:18:23 php-fpm 73087 /rc.newwanip: Resyncing OpenVPN instances for interface WAN01_CENTURYLINK. Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 64959 Aug 31 11:18:23 kernel ovpns1: link state changed to DOWN Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 kernel ovpns1: link state changed to UP Aug 31 11:18:23 php-fpm 73087 OpenVPN PID written: 98835 Aug 31 11:18:23 check_reload_status Reloading filter Aug 31 11:18:23 check_reload_status rc.newwanip starting ovpns1 Aug 31 11:18:23 php-fpm 73087 OpenVPN terminate old pid: 91710 Aug 31 11:18:23 kernel ovpns3: link state changed to DOWN Aug 31 11:18:24 kernel ovpns3: link state changed to UP Aug 31 11:18:24 php-fpm 73087 OpenVPN PID written: 20898 Aug 31 11:18:24 php-fpm 73087 /rc.newwanip: Creating rrd update script Aug 31 11:18:24 check_reload_status rc.newwanip starting ovpns3 Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: Info: starting on ovpns1. Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip: on (IP address: 192.168.31.1) (interface: []) (real interface: ovpns1). Aug 31 11:18:24 php-fpm 346 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:24 check_reload_status Reloading filter Aug 31 11:18:24 php-fpm 346 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.31.1 - Restarting packages. Aug 31 11:18:24 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: Info: starting on ovpns3. Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip: on (IP address: 192.168.32.1) (interface: []) (real interface: ovpns3). Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: rc.newwanip called with empty interface. Aug 31 11:18:25 check_reload_status Reloading filter Aug 31 11:18:25 php-fpm 86289 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.32.1 - Restarting packages. Aug 31 11:18:25 check_reload_status Starting packages Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Restarting/Starting all packages. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service avahi Aug 31 11:18:25 avahi-daemon 71257 Got SIGTERM, quitting. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 71257 Leaving mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 71257 avahi-daemon 0.7 exiting. Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service avahi Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Stopping service nut Aug 31 11:18:25 upsmon 16972 Signal 15: exiting Aug 31 11:18:25 upsd 17558 User local-monitor@::1 logged out from UPS [TrippLite_SMART1500LCD] Aug 31 11:18:25 upsd 17558 mainloop: Interrupted system call Aug 31 11:18:25 upsd 17558 Signal 15: exiting Aug 31 11:18:25 usbhid-ups 17176 Signal 15: exiting Aug 31 11:18:25 php-fpm 346 /rc.start_packages: Starting service nut Aug 31 11:18:25 upsmon 78411 Startup successful Aug 31 11:18:25 usbhid-ups 79004 Startup successful Aug 31 11:18:25 avahi-daemon 75938 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75938 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75938 avahi-daemon 0.7 starting up. Aug 31 11:18:25 avahi-daemon 75938 WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/sftp-ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Loading service file /usr/local/etc/avahi/services/ssh.service. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.4.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.3.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Joining mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1. Aug 31 11:18:25 avahi-daemon 75938 New relevant interface bce0.2.IPv4 for mDNS. Aug 31 11:18:25 avahi-daemon 75938 Network interface enumeration completed. Aug 31 11:18:25 avahi-daemon 75938 Server startup complete. Host name is Camelot.local. Local service cookie is 1381888320. Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75938 Failed to add service 'Camelot' of type '_sftp-ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/sftp-ssh.service): Not permitted Aug 31 11:18:25 avahi-daemon 75027 Found user 'avahi' (UID 558) and group 'avahi' (GID 558). Aug 31 11:18:25 avahi-daemon 75027 Successfully dropped root privileges. Aug 31 11:18:25 avahi-daemon 75027 open(/var/run/avahi-daemon//pid): File exists Aug 31 11:18:25 avahi-daemon 75027 Failed to create PID file: File exists Aug 31 11:18:26 php-fpm 73087 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 71.36.112.131 -> 71.36.127.88 - Restarting packages. -
Most of that taken when it was down?
Was something rebooted at some point in that log? When?
-
@stephenw10
The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface
Aug 31 11:18:08 ppp caught fatal signal TERM
I didn't reboot until ~11:28 or so.
This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.Thoughts?
Here are my nuclear options if I can't figure out anything else.
- Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
- Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
- Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).
-
I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC
-
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...No Mac Spoofing
System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLEThe system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive -
Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.
2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.
Steve
-
@gawainxx said in WAN interface stops working every few days.:
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
If your network setup isn't too complicated, this is what I would have done by now.
If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.
Jeff
-
@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?
Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head
.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_
If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)
-
Could a NAT rule for a Nintendo switch cause any issues?
<outbound> <mode>hybrid</mode> <rule> <source> <network>192.168.3.30/32</network> </source> <sourceport></sourceport> <descr><![CDATA[Nindento Switch|Static NAT]]></descr> <target></target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <updated> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </updated> <created> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </created> </rule>I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm
-
No, an outbound NAT rule will not be doing anything.
Traffic shaping is far more likely. Assuming it's anything config related.
Steve
-
Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.
Problem still persists.
There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.
Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?
I'm getting towards the end of my list of ideas and could desperately use some solutions.
I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?
Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xmlI'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)
-
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
-
What version of pfsense??
-
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
-
@stephenw10 said in WAN interface stops working every few days.:
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
I'm not using the ISP router for routing or dhcp atm, just handling the vlan tagged traffic to see if it has any influence...
I may have to suffer and try running a double NAT for a week or two though to see if the behaviour persists when ISP router handles traffic. -
@stephenw10 said in WAN interface stops working every few days.:
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
Interesting, I'll need to take a close look at that thread later. The webui does definately take several seconds to load when I initially try to access it while the gateway issues are occuring
-
If you are somehow hitting that still you would see high latency to the firewall itself from a LAN side client everytime you ran Status > Filter reload.
Steve
