WAN interface stops working every few days.



  • I've been having issues where my WAN interface will crap out after a few days or so.

    The behaviour is that I will have very high packet % as well as latency of 7600+MS
    already established connections (such as splashtop Remote desktop) seem unphased but any new connections and general net traffic is doa.

    Rebooting pfsense or power cycling the CenturyLink provided ONT (optical network terminal) both resolve the issue for a day or so.
    I've tried disabling and re-enabling the wan interface as well as unplugging and plugging the ethernet cable for WAN but those do not resolve the issue when it appears.

    What are some likely causes for this?



  • @gawainxx

    Try using Packet Capture, to see what's happening.



  • What would I want to focus on for the packet capture, WAN?

    also, it seems like Unplugging the WAN cable for a minute or so and plugging it back in also resolves the high latency and drops.



  • I've found this in the gateway logs if it helps any. Also Guess I'll set a syslog server back up later today.

    Aug 8 11:22:02	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr [Redacted] bind_addr [Redacted] bidentifier "WAN01_CENTURYLINK_PPPOE "
    Aug 8 13:44:52	dpinger		WAN01_CENTURYLINK_PPPOE [Redacted]: Alarm latency 544019us stddev 1364744us loss 10%
    


  • @gawainxx

    Well, you could start with what happens when you try disconnecting/reconnection the WAN cable or disabling/re-enabling the interface. I also see it's OK after waiting a minute before plugging the cable in. You could compare the differences with not waiting. That sort of thing. About 1.5 years ago, I had a problem with IPv6 on my ISP. By using Wireshark, when pfSense booted up, I was able to identify the failing equipment, by name, at my ISPs local office. After I got that resolved, I saved a normal DHCP & DHCPv6-PD sequence, so that I'd always have something to refer to, should a problem happen again. Since I was rebooting pfsense, I couldn't use Packet Capture, so I used a managed switch, configured as a data tap,with my notebook running Wireshark.



  • This post is deleted!


  • @bcruze
    These issues began after I swapped my pfsense box from an optiplex 7010 to a Dell Poweredge R210 II. it was working without any issue prior.

    As of 2 days ago this behaviour is now occuring every 12 hours or so.

    As of yesterday, I've already tried tweaking the system tunables per some suggestions for PPOE interfaces as well as BCE adapters.

    I'm going to try buying an intel gigabit nic and see if it's an issue with the broadcom onboard adapter.

    It also seems that manually setting my PPOE connection as offline, applying settings then going back and re-enabling it temporarily resolves the issue when it occurs whereas the monitoring service isn't managing to re-establish the connection.


  • Netgate Administrator

    What NICs were in the older box?

    I assume this is a PPPoE connection from your logs, unless that's another gateway?

    You could certainly try setting the gateway monitoring target to something different. Be sure it's actually the WAN and not just the target.

    Carrier grade NAT should be all at the ISP if they are using that. I'm not sure which issue you're referring to @bcruze.

    Steve



  • @stephenw10

    I'm not certain which was wan/LAN but I had two NICs, an onboard intel gigabit and a broadcom 5722.

    I still have the 5722 floating around so I'm going to see if I can adapt a full length PCIE slot bracket onto it

    I've already tried setting the gateway monitoring target to 8.8.8.8
    https://techtilt.com/fix-for-pfsense-keeps-dropping-wan-intermittently-random

    I also tried adjusting a number of settings in tunables.
    https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html
    net.isr.dispatch=deferred
    kern.ipc.nmbclusters="131072"
    hw.bce.tso_enable=0
    hw.pci.enable_msix=0

    P.S. It doesn't look like syslogging is immediately available because i let my Splunk trial license lapse... I just applied the 500MB free license but it'll take a month or a reinstall before I can access the data again.



  • Installed the 5722 going to see how it handles the PPPOe... Diddnt have a correct full length bracket so I got creative to prevent it from getting knocked out of the slot and causing a short. https://imgur.com/gallery/dbfYaLi



  • Still occurring unfortunately, would I need to run a packet capture 24/7 until an issue occurs or would there be another route?


  • Netgate Administrator

    That may not tell you much anyway.

    It's curious that rebooting the ONT corrects the issue but unplugging the cable does not. Those should be similar from pfSense's view. Obviously one resets the upstream connection too but if that were an issue then rebooting pfSense alone would not correct it.

    A short pcap made whilst the connection is bad might show something. Bad packets etc.

    It might need to be on the PPPoE parent interface though.

    Steve



  • @stephenw10 said in WAN interface stops working every few days.:

    That may not tell you much anyway.

    It's curious that rebooting the ONT corrects the issue but unplugging the cable does not. Those should be similar from pfSense's view. Obviously one resets the upstream connection too but if that were an issue then rebooting pfSense alone would not correct it.

    A short pcap made whilst the connection is bad might show something. Bad packets etc.

    It might need to be on the PPPoE parent interface though.

    Steve

    Three things have been observed to correct the issue so far.
    Rebooting pfsense
    Disconnecting the WAN if's ethernet cable for ~15 seconds then plugging it back in.
    Power Cycling the ONT

    packet loss and latency skyrockets during these events.
    I'm going to do a packet capture as well as take a close look at the PPPOE traffic the next time this happens. I'm curious to see if my WAN IP changes as well as what disabling and re-enabling PPPOE does.

    This issue began approximately 1 week after I had replaced my optiplex 7010SFF PFsense instance for the R210 II.
    There are two other things in the same timeframe which "may may potentially attribute but I'd be surprised if they were the issue"

    • Minor heat wave where temps were in the upper 90's for a few days.\
    • Unmounted ONT to physically inspect what type of optical cable it uses, It may be remotely possible that I somehow pinched the cable when returning the ONT back into it's cradle? I'm not certain whether that would manifest with these symptoms though. aside from the every 12-36 hour events pings, latency and packet loss are on par for gigabit.


  • I changed out the Broadcom NIC for an Intel one and I really, really hope this issues goes away with it.

    I did a packet capture on the WAN interface and see a lot of ttl timeouts, TCP resets or unacknowledged acks. Existing socket connections continue to work without issue but any new connection attempts have an extremely high latency and packet loss. I'm not comfortable sharing this packet capture though because it could potentially contain some authentication info which could be reverse engineered..

    I'll paste a snippet of it below.

    I'm going to see if my ISP will send me a replacement ONT so that I can cover my bases there.

    1	2020/231 17:46:10.570391	0.000000000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=123
    2	2020/231 17:46:10.570398	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=252
    3	2020/231 17:46:10.581041	0.010643000	71.36.120.123	205.251.197.145	DNS							Standard query 0x5e31 A a.teads.tv OPT
    4	2020/231 17:46:10.604454	0.023413000	71.36.120.123	182.161.72.6	DNS							Standard query 0x8f03 AAAA ns28.criteo.com OPT
    5	2020/231 17:46:10.604504	0.000050000	71.36.120.123	74.119.118.255	DNS							Standard query 0x7900 AAAA ns22.criteo.com OPT
    6	2020/231 17:46:10.604535	0.000031000	71.36.120.123	74.119.118.255	DNS							Standard query 0xa752 AAAA ns27.criteo.com OPT
    7	2020/231 17:46:10.604866	0.000331000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=128
    8	2020/231 17:46:10.610319	0.005453000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=70
    9	2020/231 17:46:10.612201	0.001882000	71.36.120.123	108.162.193.135	DNS							Standard query 0xcf67 A ns.wpopt.net OPT
    10	2020/231 17:46:10.612759	0.000558000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=42
    11	2020/231 17:46:10.616211	0.003452000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=128
    12	2020/231 17:46:10.623546	0.007335000	71.36.120.123	192.112.36.4	DNS							Standard query 0xbbf4 A wpad.britannia.local OPT
    13	2020/231 17:46:10.644779	0.021233000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=126
    14	2020/231 17:46:10.644786	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=73
    15	2020/231 17:46:10.644935	0.000149000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=627
    16	2020/231 17:46:10.644941	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=627
    17	2020/231 17:46:10.656326	0.011385000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1139
    18	2020/231 17:46:10.662098	0.005772000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=125
    19	2020/231 17:46:10.662255	0.000157000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1139
    20	2020/231 17:46:10.663498	0.001243000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    21	2020/231 17:46:10.667730	0.004232000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1139
    22	2020/231 17:46:10.671272	0.003542000	71.36.120.123	192.112.36.4	DNS							Standard query 0x06ce A local OPT
    23	2020/231 17:46:10.673319	0.002047000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1139
    24	2020/231 17:46:10.673900	0.000581000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    25	2020/231 17:46:10.678912	0.005012000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=119
    26	2020/231 17:46:10.683979	0.005067000	71.36.120.123	192.112.36.4	DNS							Standard query 0x5d0d AAAA ns-1881.awsdns-43.co.uk OPT
    27	2020/231 17:46:10.684778	0.000799000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1139
    28	2020/231 17:46:10.684785	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    29	2020/231 17:46:10.690549	0.005764000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    30	2020/231 17:46:10.695846	0.005297000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    31	2020/231 17:46:10.696177	0.000331000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    32	2020/231 17:46:10.701793	0.005616000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=114
    33	2020/231 17:46:10.701950	0.000157000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=243
    34	2020/231 17:46:10.701956	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    35	2020/231 17:46:10.707201	0.005245000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    36	2020/231 17:46:10.712448	0.005247000	71.36.120.123	192.112.36.4	DNS							Standard query 0xe812 A ns-1881.awsdns-43.co.uk OPT
    37	2020/231 17:46:10.713558	0.001110000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    38	2020/231 17:46:10.715311	0.001753000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=50
    39	2020/231 17:46:10.718780	0.003469000	71.36.120.123	204.13.251.136	DNS							Standard query 0xb24a A ns3.p29.dynect.net OPT
    40	2020/231 17:46:10.718817	0.000037000	71.36.120.123	156.154.65.210	DNS							Standard query 0xd119 A elb-ore-amz.nimbus.bitdefender.net OPT
    41	2020/231 17:46:10.719147	0.000330000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=114
    42	2020/231 17:46:10.719154	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    43	2020/231 17:46:10.724919	0.005765000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    44	2020/231 17:46:10.730536	0.005617000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    45	2020/231 17:46:10.736308	0.005772000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=114
    46	2020/231 17:46:10.736315	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    47	2020/231 17:46:10.741924	0.005609000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=46
    48	2020/231 17:46:10.741931	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1140
    49	2020/231 17:46:10.742357	0.000426000	71.36.120.123	74.125.250.87	STUN							Binding Request user: QUk4jW0q5FYFBAXl:R6ng
    50	2020/231 17:46:10.755482	0.013125000	74.125.250.87	71.36.120.123	STUN							Binding Success Response user: QUk4jW0q5FYFBAXl:R6ng XOR-MAPPED-ADDRESS: 71.36.120.123:7162
    51	2020/231 17:46:10.759071	0.003589000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=98
    52	2020/231 17:46:10.759078	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=39
    53	2020/231 17:46:10.766467	0.007389000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    54	2020/231 17:46:10.770543	0.004076000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=107
    55	2020/231 17:46:10.770549	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=702
    56	2020/231 17:46:10.771652	0.001103000	71.36.120.123	162.88.61.21	DNS							Standard query 0x78ad A ns2.p29.dynect.net OPT
    57	2020/231 17:46:10.776351	0.004699000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=77
    58	2020/231 17:46:10.783203	0.006852000	71.36.120.123	205.251.193.209	DNS							Standard query 0x4fc2 A ns-645.awsdns-16.net OPT
    59	2020/231 17:46:10.783285	0.000082000	71.36.120.123	162.88.60.21	DNS							Standard query 0xb882 A ns1.p29.dynect.net OPT
    60	2020/231 17:46:10.799214	0.015929000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=75
    61	2020/231 17:46:10.803738	0.004524000	216.239.38.10	71.36.120.123	DNS							Standard query response 0x021f A mobile-gtalk.l.google.com A 74.125.195.188 OPT
    62	2020/231 17:46:10.814661	0.010923000	192.35.51.30	71.36.120.123	DNS							Standard query response 0x43c6 A dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns3.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NSEC3 RRSIG AAAA 2001:500:90:1::136 A 208.78.70.136 OPT
    63	2020/231 17:46:10.816580	0.001919000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=69
    64	2020/231 17:46:10.817005	0.000425000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=42
    65	2020/231 17:46:10.822015	0.005010000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=76
    66	2020/231 17:46:10.825603	0.003588000	199.19.54.1	71.36.120.123	DNS							Standard query response 0x29cb A ultradns.org OPT
    67	2020/231 17:46:10.825684	0.000081000	71.36.120.123	199.19.54.1	TCP	0	0	1	0	65228		44963 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=3298201888 TSecr=0
    68	2020/231 17:46:10.827733	0.002049000	199.249.120.1	71.36.120.123	DNS							Standard query response 0xd56c AAAA ns3-06.azure-dns.org OPT
    69	2020/231 17:46:10.827740	0.000007000	65.22.162.17	71.36.120.123	TCP	0	0	1	1	65535		53 → 44907 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1452 WS=64 SACK_PERM=1 TSval=734570156 TSecr=474470959
    70	2020/231 17:46:10.827793	0.000053000	71.36.120.123	199.249.120.1	TCP	0	0	1	0	65228		44964 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=1310942176 TSecr=0
    71	2020/231 17:46:10.827892	0.000099000	199.19.53.1	71.36.120.123	TCP	0	1	1	0	0		53 → 44896 [RST] Seq=1 Win=0 Len=0
    72	2020/231 17:46:10.827899	0.000007000	192.48.79.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44909 [RST] Seq=1 Win=0 Len=0
    73	2020/231 17:46:10.829917	0.002018000	198.51.45.66	71.36.120.123	DNS							Standard query response 0x8e05 A tlx.3lift.com CNAME us-west-tlx.3lift.com CNAME dualstack.exchange-prod-582331669.us-west-1.elb.amazonaws.com OPT
    74	2020/231 17:46:10.831699	0.001782000	71.36.120.123	64.4.48.3	DNS							Standard query 0x1f8a A ns2-34.azure-dns.net OPT
    75	2020/231 17:46:10.831732	0.000033000	71.36.120.123	205.251.199.144	DNS							Standard query 0xf8f8 A ns-645.awsdns-16.net OPT
    76	2020/231 17:46:10.837963	0.006231000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=35
    77	2020/231 17:46:10.839231	0.001268000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=84
    78	2020/231 17:46:10.841102	0.001871000	199.19.53.1	71.36.120.123	TCP	0	0	1	1	65535		53 → 44916 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1452 WS=64 SACK_PERM=1 TSval=3678409839 TSecr=721965227
    79	2020/231 17:46:10.841109	0.000007000	192.5.6.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44858 [RST] Seq=1 Win=0 Len=0
    80	2020/231 17:46:10.843287	0.002178000	192.36.148.17	71.36.120.123	DNS							Standard query response 0x801c No such name A bidder.criteo.com.britannia.local OPT
    81	2020/231 17:46:10.845318	0.002031000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=246
    82	2020/231 17:46:10.845324	0.000006000	2.22.230.67	71.36.120.123	DNS							Standard query response 0x7135 A a16-65.akam.net A 23.211.132.65 OPT
    83	2020/231 17:46:10.845601	0.000277000	71.36.120.123	96.7.49.67	DNS							Standard query 0x8b78 AAAA a16-65.akam.net OPT
    84	2020/231 17:46:10.847337	0.001736000	199.7.91.13	71.36.120.123	DNS							Standard query response 0x2c86 No such name A local NSEC locker RRSIG OPT
    85	2020/231 17:46:10.849520	0.002183000	192.48.79.30	71.36.120.123	DNS							Standard query response 0x5869 A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT
    86	2020/231 17:46:10.849580	0.000060000	71.36.120.123	192.48.79.30	TCP	0	0	1	0	65228		44965 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=144028582 TSecr=0
    87	2020/231 17:46:10.850848	0.001268000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    88	2020/231 17:46:10.851470	0.000622000	2.22.230.67	71.36.120.123	DNS							Standard query response 0x31af A a5-64.akam.net A 95.100.168.64 OPT
    89	2020/231 17:46:10.851627	0.000157000	192.48.79.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44909 [RST] Seq=1 Win=0 Len=0
    90	2020/231 17:46:10.851722	0.000095000	71.36.120.123	96.7.49.67	DNS							Standard query 0x9ef7 A a5-64.akam.net OPT
    91	2020/231 17:46:10.853770	0.002048000	213.248.216.1	71.36.120.123	DNS							Standard query response 0xb526 A ns-1881.awsdns-43.co.uk NS g-ns-363.awsdns-43.co.uk NS g-ns-939.awsdns-43.co.uk NS g-ns-1518.awsdns-43.co.uk NS g-ns-1839.awsdns-43.co.uk NSEC3 RRSIG OPT
    92	2020/231 17:46:10.856579	0.002809000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    93	2020/231 17:46:10.862037	0.005458000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=87
    94	2020/231 17:46:10.862193	0.000156000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    95	2020/231 17:46:10.865180	0.002987000	71.36.120.123	216.252.166.11	DNS							Standard query 0x4a91 A ib.adnxs.com OPT
    96	2020/231 17:46:10.866755	0.001575000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    97	2020/231 17:46:10.867868	0.001113000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    98	2020/231 17:46:10.871459	0.003591000	71.36.120.123	156.154.65.210	DNS							Standard query 0x451e A elb-ore-amz.nimbus.bitdefender.net OPT
    99	2020/231 17:46:10.879433	0.007974000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=90
    100	2020/231 17:46:10.879440	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    101	2020/231 17:46:10.879589	0.000149000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    102	2020/231 17:46:10.890823	0.011234000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    103	2020/231 17:46:10.896595	0.005772000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=94
    104	2020/231 17:46:10.902368	0.005773000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=59
    105	2020/231 17:46:10.902374	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1173
    106	2020/231 17:46:10.902380	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=1174
    107	2020/231 17:46:10.908063	0.005683000	71.36.120.123	200.7.86.53	DNS							Standard query 0x760f PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa OPT
    108	2020/231 17:46:10.913699	0.005636000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=73
    109	2020/231 17:46:10.917560	0.003861000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    110	2020/231 17:46:10.919297	0.001737000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=96
    111	2020/231 17:46:10.919452	0.000155000	96.7.49.67	71.36.120.123	DNS							Standard query response 0x351a A a16-65.akam.net A 23.211.132.65 OPT
    112	2020/231 17:46:10.919458	0.000006000	81.17.242.98	71.36.120.123	ICMP							Time-to-live exceeded (Time to live exceeded in transit)
    113	2020/231 17:46:10.919465	0.000007000	81.17.242.98	71.36.120.123	ICMP							Time-to-live exceeded (Time to live exceeded in transit)
    114	2020/231 17:46:10.919717	0.000252000	71.36.120.123	23.211.133.67	DNS							Standard query 0xb7b3 A a16-65.akam.net OPT
    115	2020/231 17:46:10.921607	0.001890000	96.7.49.67	71.36.120.123	DNS							Standard query response 0x8d3e AAAA use2.akam.net SOA internal.akam.net OPT
    116	2020/231 17:46:10.921867	0.000260000	71.36.120.123	23.211.133.67	DNS							Standard query 0x2bb5 AAAA use2.akam.net OPT
    117	2020/231 17:46:10.923758	0.001891000	199.253.182.182	71.36.120.123	DNS							Standard query response 0xfdd1 PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa NS r.arin.net NS u.arin.net NS x.arin.net NS y.arin.net NS z.arin.net NS arin.authdns.ripe.net DS RRSIG OPT
    118	2020/231 17:46:10.925005	0.001247000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=697
    119	2020/231 17:46:10.925786	0.000781000	199.19.56.1	71.36.120.123	DNS							Standard query response 0x3f87 A ultradns.org OPT
    120	2020/231 17:46:10.925846	0.000060000	71.36.120.123	199.19.56.1	TCP	0	0	1	0	65228		44966 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=923319125 TSecr=0
    121	2020/231 17:46:10.927895	0.002049000	198.51.44.2	71.36.120.123	DNS							Standard query response 0xb0f4 A prebid.appnexusgslb.net A 68.67.129.85 OPT
    122	2020/231 17:46:10.929922	0.002027000	65.22.163.17	71.36.120.123	DNS							Standard query response 0x4662 A dmx.districtm.io OPT
    123	2020/231 17:46:10.930703	0.000781000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=39
    124	2020/231 17:46:10.931952	0.001249000	23.211.133.67	71.36.120.123	DNS							Standard query response 0x4d8f AAAA a28-67.akam.net SOA internal.akam.net OPT
    125	2020/231 17:46:10.932073	0.000121000	71.36.120.123	184.85.248.67	DNS							Standard query 0x2361 AAAA a28-67.akam.net OPT
    126	2020/231 17:46:10.934120	0.002047000	192.36.148.17	71.36.120.123	DNS							Standard query response 0x940b DNSKEY <Root> OPT
    127	2020/231 17:46:10.936146	0.002026000	23.211.133.67	71.36.120.123	DNS							Standard query response 0x0d65 A a5-64.akam.net A 95.100.168.64 OPT
    128	2020/231 17:46:10.936393	0.000247000	71.36.120.123	95.101.36.67	DNS							Standard query 0x51e0 AAAA a5-64.akam.net OPT
    129	2020/231 17:46:10.942964	0.006571000	192.48.79.30	71.36.120.123	DNS							Standard query response 0x4b33 A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT
    130	2020/231 17:46:10.943024	0.000060000	71.36.120.123	192.48.79.30	TCP	0	0	1	0	65228		44967 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2408885028 TSecr=0
    131	2020/231 17:46:10.945072	0.002048000	199.254.48.1	71.36.120.123	DNS							Standard query response 0xd584 AAAA ns4-06.azure-dns.info OPT
    132	2020/231 17:46:10.947099	0.002027000	199.254.48.1	71.36.120.123	DNS							Standard query response 0xb8a3 A ns4-06.azure-dns.info OPT
    133	2020/231 17:46:10.947881	0.000782000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=85
    134	2020/231 17:46:10.949130	0.001249000	199.253.182.182	71.36.120.123	DNS							Standard query response 0xf9cc PTR 4.d.3.2.0.4.f.2.b.0.d.3.0.0.7.2.e.c.5.0.9.1.6.f.0.2.c.f.7.0.6.2.ip6.arpa NS r.arin.net NS u.arin.net NS x.arin.net NS y.arin.net NS z.arin.net NS arin.authdns.ripe.net DS RRSIG OPT
    135	2020/231 17:46:10.950288	0.001158000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=46
    136	2020/231 17:46:10.951244	0.000956000	182.161.72.6	71.36.120.123	DNS							Standard query response 0x1f80 AAAA ns26.criteo.com SOA ns23.criteo.com OPT
    137	2020/231 17:46:10.951251	0.000007000	8.8.8.8	71.36.120.123	ICMP							Echo (ping) reply    id=0x0e8f, seq=389/34049, ttl=118
    138	2020/231 17:46:10.953275	0.002024000	172.217.14.196	71.36.120.123	TCP	0	0	1	1	65535		80 → 25037 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=2310718172 TSecr=149079597 WS=256
    139	2020/231 17:46:10.953430	0.000155000	192.48.79.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44909 [RST] Seq=1 Win=0 Len=0
    140	2020/231 17:46:10.955459	0.002029000	96.7.49.67	71.36.120.123	DNS							Standard query response 0x6fcb A as-sec.casalemedia.com CNAME as-sec.casalemedia.com.edgekey.net OPT
    141	2020/231 17:46:10.959517	0.004058000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=84
    142	2020/231 17:46:10.959829	0.000312000	192.48.79.30	71.36.120.123	DNS							Standard query response 0x4c0b A nsone.net NS dns1.p01.nsone.net NS dns2.p01.nsone.net NS dns3.p01.nsone.net NS dns4.p01.nsone.net NSEC3 RRSIG NSEC3 A 198.51.44.1 A 198.51.45.1 OPT
    143	2020/231 17:46:10.959895	0.000066000	71.36.120.123	192.48.79.30	TCP	0	0	1	0	65228		44968 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=3466359485 TSecr=0
    144	2020/231 17:46:10.961788	0.001893000	192.112.36.4	71.36.120.123	DNS							Standard query response 0xcbc7 No such name A wpad.britannia.local OPT
    145	2020/231 17:46:10.962221	0.000433000	71.36.120.123	192.12.94.30	DNS							Standard query 0x61b2 A appnexusgslb.com OPT
    146	2020/231 17:46:10.963956	0.001735000	96.7.49.67	71.36.120.123	DNS							Standard query response 0x1a52 A a16-65.akam.net A 23.211.132.65 OPT
    147	2020/231 17:46:10.964207	0.000251000	71.36.120.123	95.101.36.67	DNS							Standard query 0xf4ef A a16-65.akam.net OPT
    148	2020/231 17:46:10.967801	0.003594000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=42
    149	2020/231 17:46:10.982330	0.014529000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=79
    150	2020/231 17:46:10.982337	0.000007000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=327
    151	2020/231 17:46:10.985450	0.003113000	172.217.14.196	71.36.120.123	TCP	0	0	1	1	65535		443 → 39665 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=2778687698 TSecr=149079006 WS=256
    152	2020/231 17:46:10.985456	0.000006000	172.217.14.195	71.36.120.123	TCP	0	0	1	1	65535		80 → 6268 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 TSval=1227369459 TSecr=149079007 WS=256
    153	2020/231 17:46:10.987478	0.002022000	156.154.101.3	71.36.120.123	DNS							Standard query response 0x1d19 AAAA nsc.nic.uk SOA dns1.nic.uk OPT
    154	2020/231 17:46:10.989118	0.001640000	71.36.120.123	173.245.59.135	DNS							Standard query 0x4482 A ns.wpopt.net OPT
    155	2020/231 17:46:10.989148	0.000030000	71.36.120.123	172.64.33.135	DNS							Standard query 0x5a54 A ns.wpopt.net OPT
    156	2020/231 17:46:10.989633	0.000485000	96.7.49.67	71.36.120.123	DNS							Standard query response 0xbb37 AAAA a9-67.akam.net AAAA 2a02:26f0:117::43 OPT
    157	2020/231 17:46:10.989875	0.000242000	71.36.120.123	184.85.248.67	DNS							Standard query 0xbdfc AAAA a9-67.akam.net OPT
    158	2020/231 17:46:10.990877	0.001002000	71.36.120.123	172.217.14.196	TCP	0	1	1	1	343		25037 → 80 [ACK] Seq=1 Ack=1 Win=343 Len=0 TSval=149080659 TSecr=2310718172
    159	2020/231 17:46:10.990892	0.000015000	71.36.120.123	172.217.14.196	TCP	0	882	882	1076	354		[TCP ACKed unseen segment] 39665 → 443 [ACK] Seq=882 Ack=1076 Win=354 Len=0 TSval=149080659 TSecr=2778687985 SLE=0 SRE=1
    160	2020/231 17:46:10.991098	0.000206000	71.36.120.123	172.217.14.195	TCP	0	229	229	103	343		[TCP ACKed unseen segment] 6268 → 80 [ACK] Seq=229 Ack=103 Win=343 Len=0 TSval=149080659 TSecr=1227369723 SLE=0 SRE=1
    161	2020/231 17:46:10.991362	0.000264000	71.36.120.123	172.217.14.196	HTTP	207	1	208	1	343	207	GET /gen_204 HTTP/1.1 
    162	2020/231 17:46:10.993567	0.002205000	176.32.99.148	71.36.120.123	TLSv1.2	46	1	47	1	2188	46	Application Data
    163	2020/231 17:46:10.996531	0.002964000	172.217.14.196	71.36.120.123	TCP	0	1	1	208	66816		80 → 25037 [ACK] Seq=1 Ack=208 Win=66816 Len=0 TSval=2310719898 TSecr=149080659
    164	2020/231 17:46:10.999340	0.002809000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=79
    165	2020/231 17:46:11.000038	0.000698000	71.36.120.123	176.32.99.148	TLSv1.2	46	1	47	47	8209	46	Application Data
    166	2020/231 17:46:11.002999	0.002961000	71.36.120.123	198.51.45.2	DNS							Standard query 0x3b38 A tlx.3lift.com OPT
    167	2020/231 17:46:11.005044	0.002045000	192.48.79.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44909 [RST] Seq=1 Win=0 Len=0
    168	2020/231 17:46:11.007073	0.002029000	192.112.36.4	71.36.120.123	DNS							Standard query response 0xd94a No such name A local OPT
    169	2020/231 17:46:11.008634	0.001561000	172.217.14.196	71.36.120.123	HTTP	314	1	315	208	66816	314	HTTP/1.1 204 No Content 
    170	2020/231 17:46:11.009259	0.000625000	192.112.36.4	71.36.120.123	DNS							Standard query response 0xcd80 AAAA ns-1881.awsdns-43.co.uk OPT
    171	2020/231 17:46:11.010640	0.001381000	71.36.120.123	172.217.14.196	TCP	0	208	208	315	347		25037 → 80 [ACK] Seq=208 Ack=315 Win=347 Len=0 TSval=149080664 TSecr=2310719910
    172	2020/231 17:46:11.011283	0.000643000	23.211.133.67	71.36.120.123	DNS							Standard query response 0x8337 A a16-65.akam.net A 23.211.132.65 OPT
    173	2020/231 17:46:11.011532	0.000249000	71.36.120.123	95.100.173.67	DNS							Standard query 0x5f5c AAAA a16-65.akam.net OPT
    174	2020/231 17:46:11.013225	0.001693000	71.36.120.123	172.217.14.196	TCP	0	208	209	315	347		25037 → 80 [FIN, ACK] Seq=208 Ack=315 Win=347 Len=0 TSval=149080665 TSecr=2310719910
    175	2020/231 17:46:11.013400	0.000175000	2.22.230.67	71.36.120.123	DNS							Standard query response 0x0068 AAAA a22-67.akam.net SOA internal.akam.net OPT
    176	2020/231 17:46:11.013406	0.000006000	81.17.242.98	71.36.120.123	ICMP							Time-to-live exceeded (Time to live exceeded in transit)
    177	2020/231 17:46:11.013535	0.000129000	71.36.120.123	95.100.173.67	DNS							Standard query 0x2cbf AAAA a22-67.akam.net OPT
    178	2020/231 17:46:11.015427	0.001892000	192.112.36.4	71.36.120.123	DNS							Standard query response 0x5271 A ns-1881.awsdns-43.co.uk OPT
    179	2020/231 17:46:11.018265	0.002838000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=42
    180	2020/231 17:46:11.019300	0.001035000	71.36.120.123	8.8.8.8	ICMP							Echo (ping) request  id=0x0e8f, seq=396/35841, ttl=64 (no response found!)
    181	2020/231 17:46:11.019344	0.000044000	71.36.120.123	192.12.94.30	DNS							Standard query 0xf06c A ns27.domaincontrol.com OPT
    182	2020/231 17:46:11.019378	0.000034000	71.36.120.123	192.12.94.30	DNS							Standard query 0x0b0c AAAA ns27.domaincontrol.com OPT
    183	2020/231 17:46:11.019410	0.000032000	71.36.120.123	192.12.94.30	DNS							Standard query 0xaaef AAAA ns28.domaincontrol.com OPT
    184	2020/231 17:46:11.022237	0.002827000	156.154.65.210	71.36.120.123	DNS							Standard query response 0x8827 A elb-ore-amz.nimbus.bitdefender.net CNAME kube-nimbus-471965604.us-west-2.elb.amazonaws.com OPT
    185	2020/231 17:46:11.022391	0.000154000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=80
    186	2020/231 17:46:11.024265	0.001874000	204.13.251.136	71.36.120.123	DNS							Standard query response 0x4bf1 A ns3.p29.dynect.net A 208.78.71.29 NS ns3.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net OPT
    187	2020/231 17:46:11.026294	0.002029000	162.88.61.21	71.36.120.123	DNS							Standard query response 0x4046 A ns2.p29.dynect.net A 204.13.250.29 OPT
    188	2020/231 17:46:11.026762	0.000468000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=66
    189	2020/231 17:46:11.028484	0.001722000	192.58.128.30	71.36.120.123	DNS							Standard query response 0x47b3 A biz NS k.gtld.biz NS f.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz DS DS RRSIG OPT
    190	2020/231 17:46:11.030513	0.002029000	192.5.5.241	71.36.120.123	DNS							Standard query response 0x724c A biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT
    191	2020/231 17:46:11.030578	0.000065000	71.36.120.123	192.5.5.241	TCP	0	0	1	0	65228		44969 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2411125480 TSecr=0
    192	2020/231 17:46:11.032627	0.002049000	192.5.5.241	71.36.120.123	DNS							Standard query response 0x847d A e.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT
    193	2020/231 17:46:11.032683	0.000056000	71.36.120.123	192.5.5.241	TCP	0	0	1	0	65228		44970 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=259106889 TSecr=0
    194	2020/231 17:46:11.036916	0.004233000	204.13.250.136	71.36.120.123	DNS							Standard query response 0xfdbf A ns2.p29.dynect.net A 204.13.250.29 NS ns3.dynamicnetworkservices.net NS ns2.dynamicnetworkservices.net NS ns7.dynamicnetworkservices.net NS ns1.dynamicnetworkservices.net NS ns6.dynamicnetworkservices.net NS ns4.dynamicnetworkservices.net NS ns5.dynamicnetworkservices.net OPT
    195	2020/231 17:46:11.037107	0.000191000	71.36.120.123	208.78.71.136	DNS							Standard query 0x9ddf A ns2.p29.dynect.net OPT
    196	2020/231 17:46:11.038999	0.001892000	205.251.195.18	71.36.120.123	DNS							Standard query response 0x4d9e A ns-645.awsdns-16.net A 205.251.194.133 NS g-ns-1360.awsdns-16.net NS g-ns-1936.awsdns-16.net NS g-ns-465.awsdns-16.net NS g-ns-786.awsdns-16.net A 205.251.197.80 AAAA 2600:9000:5305:5000::1 A 205.251.199.144 AAAA 2600:9000:5307:9000::1 A 205.251.193.209 AAAA 2600:9000:5301:d100::1 A 205.251.195.18 AAAA 2600:9000:5303:1200::1 OPT
    197	2020/231 17:46:11.039007	0.000008000	192.35.51.30	71.36.120.123	TCP	0	1	1	0	0		53 → 44915 [RST] Seq=1 Win=0 Len=0
    198	2020/231 17:46:11.039467	0.000460000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=83
    199	2020/231 17:46:11.039473	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=620
    200	2020/231 17:46:11.041027	0.001554000	192.42.93.30	71.36.120.123	DNS							Standard query response 0x4e36 A amplitude.com NS ns-579.awsdns-08.net NS ns-260.awsdns-32.com NS ns-1262.awsdns-29.org NS ns-1942.awsdns-50.co.uk NSEC3 RRSIG A 205.251.193.4 OPT
    201	2020/231 17:46:11.043056	0.002029000	198.97.190.53	71.36.120.123	DNS							Standard query response 0x47e6 A biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT
    202	2020/231 17:46:11.043114	0.000058000	71.36.120.123	198.97.190.53	TCP	0	0	1	0	65228		44971 → 53 [SYN] Seq=0 Win=65228 Len=0 MSS=1452 WS=128 SACK_PERM=1 TSval=2608320456 TSecr=0
    203	2020/231 17:46:11.045161	0.002047000	162.88.60.21	71.36.120.123	DNS							Standard query response 0x346c A ns1.p29.dynect.net A 208.78.70.29 OPT
    204	2020/231 17:46:11.047467	0.002306000	71.36.120.123	216.239.34.10	DNS							Standard query 0xa45d A mobile-gtalk.l.google.com OPT
    205	2020/231 17:46:11.050920	0.003453000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=79
    206	2020/231 17:46:11.050926	0.000006000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=107
    207	2020/231 17:46:11.053726	0.002800000	205.251.193.209	71.36.120.123	DNS							Standard query response 0x79d3 A ns-645.awsdns-16.net A 205.251.194.133 NS g-ns-1360.awsdns-16.net NS g-ns-1936.awsdns-16.net NS g-ns-465.awsdns-16.net NS g-ns-786.awsdns-16.net A 205.251.197.80 AAAA 2600:9000:5305:5000::1 A 205.251.199.144 AAAA 2600:9000:5307:9000::1 A 205.251.193.209 AAAA 2600:9000:5301:d100::1 A 205.251.195.18 AAAA 2600:9000:5303:1200::1 OPT
    208	2020/231 17:46:11.055755	0.002029000	156.154.65.210	71.36.120.123	DNS							Standard query response 0x0f00 A elb-ore-amz.nimbus.bitdefender.net CNAME kube-nimbus-471965604.us-west-2.elb.amazonaws.com OPT
    209	2020/231 17:46:11.057944	0.002189000	64.4.48.1	71.36.120.123	DNS							Standard query response 0x3e3f A ns2-34.azure-dns.net A 150.171.16.34 OPT
    210	2020/231 17:46:11.059971	0.002027000	205.251.194.68	71.36.120.123	DNS							Standard query response 0x3039 AAAA ns-38.awsdns-04.com AAAA 2600:9000:5300:2600::1 NS g-ns-1156.awsdns-04.com NS g-ns-1732.awsdns-04.com NS g-ns-5.awsdns-04.com NS g-ns-580.awsdns-04.com A 205.251.196.132 AAAA 2600:9000:5304:8400::1 A 205.251.198.196 AAAA 2600:9000:5306:c400::1 A 205.251.192.5 AAAA 2600:9000:5300:500::1 A 205.251.194.68 AAAA 2600:9000:5302:4400::1 OPT
    211	2020/231 17:46:11.062155	0.002184000	2.22.230.67	71.36.120.123	DNS							Standard query response 0x6d3c A a9-67.akam.net A 184.85.248.67 OPT
    212	2020/231 17:46:11.062411	0.000256000	71.36.120.123	95.100.173.67	DNS							Standard query 0xd1e4 AAAA a9-67.akam.net OPT
    213	2020/231 17:46:11.064145	0.001734000	43.230.48.1	71.36.120.123	DNS							Standard query response 0xa2b0 AAAA nsd.nic.uk SOA dns1.nic.uk OPT
    214	2020/231 17:46:11.066017	0.001872000	74.125.250.87	71.36.120.123	UDP							19305 → 7162 Len=66
    215	2020/231 17:46:11.066176	0.000159000	198.97.190.53	71.36.120.123	DNS							Standard query response 0x09cf A a.gtld.biz NS a.gtld.biz NS b.gtld.biz NS c.gtld.biz NS e.gtld.biz NS f.gtld.biz NS k.gtld.biz DS DS RRSIG OPT
    216	2020/231 17:46:11.067344	0.001168000	71.36.120.123	74.125.250.87	UDP							7162 → 19305 Len=42
    217	2020/231 17:46:11.068301	0.000957000	216.252.166.10	71.36.120.123	DNS							Standard query response 0xd535 A ib.adnxs.com CNAME g.geogslb.com NS ns1.gslb.com NS ns2.gslb.com
    218	2020/231 17:46:11.068307	0.000006000	81.17.242.98	71.36.120.123	ICMP							Time-to-live exceeded (Time to live exceeded in transit)
    

  • Netgate Administrator

    Hmm, that sure starts to looks like an upstream routing issue.

    What is sending that ICMP TTL exceeded response? What was the target?

    Steve



  • @stephenw10 target was the google DNS server, 8.8.8.8

    So far this week, the issue has manifested like clockwork almost every day between 10:50 and 11:15 AM. with one occasion where it also reoccurred near noon as well.

    I've contacted my ISP and they beleive they saw some up line issues and have a tech coming out next week...

    I'm getting very tiered of this issue very fast.



  • ISP replaced the ONT and I had been problem free until today when the behaviour appeared again..

    I tried to do a tracert and every hop diddnt response and the last 8.8.8.8 had a response time of 1248ms

    I was able to restore my connection by going to status>interfaces and then disconnecting and recconecting the WAN PPOE.

    Could use some guidance on troubleshooting PPOE issues as well as reccomendations on a scripted workaround to automatically restart it if non responsive after a period of time.


  • Netgate Administrator

    You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
    It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?

    You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.

    Steve



  • @stephenw10 said in WAN interface stops working every few days.:

    You never said what in the route is sending TTL exceeded replies annd what the acrual message is. That's usually a sign there's a routing loop.
    It looks like 81.17.242.98 and sending the replies back to 71.36.120.123 which I assume was your WAN IP at that time. What is 81.17.242.98 though? Something at your ISP?

    You can configure a PPPoE connection to reset at, say, 6am everyday. That will likely prevent this if it doesn't fail more often than that. Though it should not be required.

    Steve

    I'll have to grab that info the next time this behavior occurs, which specific info would I want to grab in this case?

    Not sure on that specific AP, it was likely picking up traffic from some random device on my network.

    Here's my config related to my PPOE wan if that helps any.

    	<wan>
    		<if>pppoe0</if>
    		<blockbogons></blockbogons>
    		<descr><![CDATA[WAN01_CenturyLink]]></descr>
    		<alias-address></alias-address>
    		<alias-subnet>32</alias-subnet>
    		<spoofmac></spoofmac>
    		<blockpriv></blockpriv>
    		<enable></enable>
    		<ipaddr>pppoe</ipaddr>
    	</wan>
    	<vlan>
    		<if>igb0</if>
    		<tag>201</tag>
    		<pcp></pcp>
    		<descr><![CDATA[WAN_01_VLAN201]]></descr>
    		<vlanif>igb0.201</vlanif>
    	</vlan>
    <ppps>
    	<ppp>
    		<ptpid>0</ptpid>
    		<type>pppoe</type>
    		<if>pppoe0</if>
    		<ports>igb0.201</ports>
    		<username><![CDATA[REDACTED@centurylink.net]]></username>
    		<password><![CDATA[REDACTED]]></password>
    		<bandwidth></bandwidth>
    		<mtu></mtu>
    		<mru></mru>
    		<mrru></mrru>
    	</ppp>
    </ppps>
    <gateways>
    	<gateway_item>
    		<interface>wan</interface>
    		<gateway>dynamic</gateway>
    		<name>WAN01_CENTURYLINK_PPPOE</name>
    		<weight>1</weight>
    		<ipprotocol>inet</ipprotocol>
    		<descr><![CDATA[Interface WAN01_CENTURYLINK_PPPOE Gateway]]></descr>
    		<monitor>8.8.8.8</monitor>
    	</gateway_item>
    	<defaultgw4>WAN01_CENTURYLINK_PPPOE</defaultgw4>
    	<defaultgw6>-</defaultgw6>
    </gateways>

  • Netgate Administrator

    Nothing unusual there.

    You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.

    Steve



  • @stephenw10 said in WAN interface stops working every few days.:

    Nothing unusual there.

    You can set a periodic reset as I said. You might try that to see if it does prevent the issue happening during the day.

    Steve

    It's unfortunately sometimes occurs more frequently then that. Last event was yesterday around ~1pm and it reoccured a short bit ago around 9:20am today.

    I was not able to get the connection back this time by disconnecting and reconnecting the PPOE cconnection, ended up restarting PFsense.

    Next step will likely be for me to disable snort for atleast a week or until the issue returns to see if the behaviour reappears.

    I'm kind of grasping at straws right now though.....

    ------------ System logs from time period ---------

    Aug 31 09:10:20	snort	67712	[1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.77.227:33798 -> 71.36.122.177:443
    Aug 31 09:10:57	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 183.131.3.210:58864 -> 71.36.122.177:1433
    Aug 31 09:11:25	snort	67712	[1:2403368:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 35 [Classification: Misc Attack] [Priority: 2] {TCP} 51.161.12.231:32767 -> 71.36.122.177:8545
    Aug 31 09:13:13	snort	67712	[1:2403448:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 75 [Classification: Misc Attack] [Priority: 2] {TCP} 89.248.168.157:37856 -> 71.36.122.177:41065
    Aug 31 09:14:38	snort	67712	[1:2403458:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 80 [Classification: Misc Attack] [Priority: 2] {TCP} 92.63.197.55:40327 -> 71.36.122.177:3377
    Aug 31 09:15:07	snort	67712	[1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.56.238:55872 -> 71.36.122.177:5900
    Aug 31 09:16:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.20:57576 -> 71.36.122.177:3345
    Aug 31 09:16:14	rc.gateway_alarm	27046	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:506.622ms RTTsd:787.570ms Loss:0%)
    Aug 31 09:16:14	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:16:14	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:16:14	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:16:14	check_reload_status		Reloading filter
    Aug 31 09:16:15	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:16:15	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:17:07	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.3:55957 -> 71.36.122.177:3310
    Aug 31 09:17:07	snort	67712	[1:2403460:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 81 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291
    Aug 31 09:17:07	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 94.102.51.17:51800 -> 71.36.122.177:7291
    Aug 31 09:17:22	rc.gateway_alarm	11126	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4120.023ms RTTsd:1799.455ms Loss:22%)
    Aug 31 09:17:22	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:17:22	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:17:22	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:17:22	check_reload_status		Reloading filter
    Aug 31 09:17:23	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:17:23	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:17:27	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852
    Aug 31 09:17:27	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.142:45646 -> 71.36.122.177:17852
    Aug 31 09:17:35	rc.gateway_alarm	61503	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3703.111ms RTTsd:2201.113ms Loss:11%)
    Aug 31 09:17:35	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:17:35	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:17:35	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:17:35	check_reload_status		Reloading filter
    Aug 31 09:17:36	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:17:36	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:17:38	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.102:47924 -> 71.36.122.177:26098
    Aug 31 09:18:31	snort	67712	[1:2403424:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 63 [Classification: Misc Attack] [Priority: 2] {TCP} 78.108.177.54:26525 -> 71.36.122.177:8080
    Aug 31 09:18:32	rc.gateway_alarm	50465	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:0 RTT:310.577ms RTTsd:435.870ms Loss:0%)
    Aug 31 09:18:32	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:18:32	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:18:32	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:18:32	check_reload_status		Reloading filter
    Aug 31 09:18:33	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:18:34	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:18:57	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.91:45181 -> 71.36.122.177:33355
    Aug 31 09:19:52	snort	67712	[1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.113:42826 -> 71.36.122.177:3391
    Aug 31 09:20:03	snort	67712	[1:2400005:2773] ET DROP Spamhaus DROP Listed Traffic Inbound group 6 [Classification: Misc Attack] [Priority: 2] {TCP} 103.215.80.70:6000 -> 71.36.122.177:6780
    Aug 31 09:20:44	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573
    Aug 31 09:20:44	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.47:50206 -> 71.36.122.177:15573
    Aug 31 09:22:03	snort	67712	[1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060
    Aug 31 09:22:03	snort	67712	[1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 193.203.14.202:5311 -> 71.36.122.177:5060
    Aug 31 09:22:27	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433
    Aug 31 09:22:29	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 103.48.25.131:63333 -> 71.36.122.177:1433
    Aug 31 09:24:01	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.4:55935 -> 71.36.122.177:835
    Aug 31 09:24:26	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.8:55838 -> 71.36.122.177:4004
    Aug 31 09:26:21	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124
    Aug 31 09:26:21	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.24:43406 -> 71.36.122.177:22124
    Aug 31 09:27:05	snort	67712	[1:2403406:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 54 [Classification: Misc Attack] [Priority: 2] {TCP} 62.171.161.187:43973 -> 71.36.122.177:81
    Aug 31 09:28:11	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606
    Aug 31 09:28:11	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.151:51260 -> 71.36.122.177:37606
    Aug 31 09:28:47	snort	67712	[1:2403429:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 65 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.212:48824 -> 71.36.122.177:49154
    Aug 31 09:28:52	rc.gateway_alarm	69361	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:502.168ms RTTsd:986.015ms Loss:0%)
    Aug 31 09:28:52	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:28:52	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:28:52	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:28:52	check_reload_status		Reloading filter
    Aug 31 09:28:53	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:28:53	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:28:56	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.53:57620 -> 71.36.122.177:6357
    Aug 31 09:29:02	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139
    Aug 31 09:29:02	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.46:52212 -> 71.36.122.177:15139
    Aug 31 09:29:12	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856
    Aug 31 09:29:12	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.141:45527 -> 71.36.122.177:17856
    Aug 31 09:29:44	snort	67712	[1:2403419:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 60 [Classification: Misc Attack] [Priority: 2] {UDP} 71.6.158.166:32064 -> 71.36.122.177:389
    Aug 31 09:30:04	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867
    Aug 31 09:30:04	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.148:44932 -> 71.36.122.177:17867
    Aug 31 09:30:14	snort	67712	[1:2011716:4] ET SCAN Sipvicious User-Agent Detected (friendly-scanner) [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060
    Aug 31 09:30:14	snort	67712	[1:2008578:6] ET SCAN Sipvicious Scan [Classification: Attempted Information Leak] [Priority: 2] {UDP} 51.89.217.179:5072 -> 71.36.122.177:5060
    Aug 31 09:30:26	snort	67712	[1:2403452:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 77 [Classification: Misc Attack] [Priority: 2] {TCP} 91.229.112.11:48084 -> 71.36.122.177:10552
    Aug 31 09:31:13	rc.gateway_alarm	93277	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4050.647ms RTTsd:1954.397ms Loss:21%)
    Aug 31 09:31:13	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:31:13	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:31:13	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:31:13	check_reload_status		Reloading filter
    Aug 31 09:31:14	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:31:14	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:31:23	rc.gateway_alarm	78618	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4322.346ms RTTsd:1981.268ms Loss:14%)
    Aug 31 09:31:23	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:31:23	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:31:23	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:31:23	check_reload_status		Reloading filter
    Aug 31 09:31:24	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:31:24	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:32:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.174:44528 -> 71.36.122.177:33339
    Aug 31 09:32:41	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872
    Aug 31 09:32:41	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.143:44684 -> 71.36.122.177:17872
    Aug 31 09:32:58	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015
    Aug 31 09:32:58	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.12:41414 -> 71.36.122.177:62015
    Aug 31 09:33:17	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 195.54.167.94:45253 -> 71.36.122.177:33384
    Aug 31 09:33:56	snort	67712	[1:2403431:59789] ET CINS Active Threat Intelligence Poor Reputation IP UDP group 66 [Classification: Misc Attack] [Priority: 2] {UDP} 80.82.77.245:44258 -> 71.36.122.177:120
    Aug 31 09:34:18	snort	67712	[1:2403436:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 69 [Classification: Misc Attack] [Priority: 2] {TCP} 83.97.20.35:48991 -> 71.36.122.177:6664
    Aug 31 09:34:28	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.21:56468 -> 71.36.122.177:22979
    Aug 31 09:35:11	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.230:40882 -> 71.36.122.177:3997
    Aug 31 09:35:15	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.176.27.14:49426 -> 71.36.122.177:26187
    Aug 31 09:35:25	snort	67712	[1:2403454:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 78 [Classification: Misc Attack] [Priority: 2] {TCP} 91.240.118.60:53196 -> 71.36.122.177:4184
    Aug 31 09:35:38	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139
    Aug 31 09:35:38	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.10:57057 -> 71.36.122.177:27139
    Aug 31 09:36:18	snort	67712	[1:2010935:3] ET SCAN Suspicious inbound to MSSQL port 1433 [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 124.114.177.237:10566 -> 71.36.122.177:1433
    Aug 31 09:36:35	snort	67712	[1:2403492:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 97 [Classification: Misc Attack] [Priority: 2] {TCP} 106.13.48.122:57394 -> 71.36.122.177:774
    Aug 31 09:36:39	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548
    Aug 31 09:36:39	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.5:42685 -> 71.36.122.177:5548
    Aug 31 09:36:59	snort	67712	[1:2403428:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 65 [Classification: Misc Attack] [Priority: 2] {TCP} 80.82.65.74:58855 -> 71.36.122.177:6000
    Aug 31 09:37:09	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956
    Aug 31 09:37:09	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.45:50080 -> 71.36.122.177:14956
    Aug 31 09:37:11	snort	67712	[1:2403344:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23 [Classification: Misc Attack] [Priority: 2] {TCP} 45.145.66.22:56634 -> 71.36.122.177:33046
    Aug 31 09:37:31	snort	67712	[1:2403342:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 22 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547
    Aug 31 09:37:31	snort	67712	[1:2402000:5651] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 45.129.33.15:56776 -> 71.36.122.177:3547
    Aug 31 09:37:33	rc.gateway_alarm	53811	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4054.569ms RTTsd:2049.170ms Loss:21%)
    Aug 31 09:37:33	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 09:37:33	check_reload_status		Restarting ipsec tunnels
    Aug 31 09:37:33	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 09:37:33	check_reload_status		Reloading filter
    Aug 31 09:37:34	php-fpm		/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 09:37:34	php-fpm		/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 09:37:48	snort	67712	[1:2403372:59789] ET CINS Active Threat Intelligence Poor Reputation IP TCP group 37 [Classification: Misc Attack] [Priority: 2] {TCP} 54.36.109.237:50023 -> 71.36.122.177:8443
    

    ---------- Gateway logs from time period ------------------

    Aug 30 13:32:43	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 290791us stddev 369179us loss 0%
    Aug 31 09:16:14	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 506622us stddev 787570us loss 0%
    Aug 31 09:17:22	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4120023us stddev 1799455us loss 22%
    Aug 31 09:17:35	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 3703111us stddev 2201113us loss 11%
    Aug 31 09:18:32	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Clear latency 310577us stddev 435870us loss 0%
    Aug 31 09:28:52	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 502168us stddev 986015us loss 0%
    Aug 31 09:31:13	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4050647us stddev 1954397us loss 21%
    Aug 31 09:31:23	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4322346us stddev 1981268us loss 14%
    Aug 31 09:37:33	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 4054569us stddev 2049170us loss 21%
    Aug 31 09:40:13	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 97.120.6.183 identifier "WAN01_CENTURYLINK_PPPOE "
    Aug 31 09:40:30	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 507360us stddev 451625us loss 0%
    Aug 31 09:40:36	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 599186us stddev 671081us loss 22%
    Aug 31 09:40:46	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1544978us stddev 1669473us loss 11%
    Aug 31 09:41:13	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 1609645us stddev 1562133us loss 21%
    Aug 31 09:41:18	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 75.164.130.187 identifier "WAN01_CENTURYLINK_PPPOE "
    Aug 31 09:41:30	dpinger		WAN01_CENTURYLINK_PPPOE 8.8.4.4: Alarm latency 589734us stddev 844410us loss 14%
    

    --- End logs----

    I'll need to look closer at the PPP logs the next time this occurs, They were unfortunately flooded out when I restarted pfsense.
    I've also been collecting data into Splunk, I'll need to go through that and set up filters when I have time today.


  • Netgate Administrator

    Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?



  • @stephenw10 said in WAN interface stops working every few days.:

    Yeah the gateway logs look terrible. It's not failing on each of those events? Just very bad latency and/or packet loss?

    And it just occurred AGAIN, approx 2 hours later.
    Restarted the router another time, This is getting very old and frustrating very fast.
    I would love any guidance I can get on next steps.

    Bullet Points I can think of

    • This behavior began a week or so after I switched from A Dell Optiplex 7010 SFF to a Poweredge R210
    • Restarting PfSense or the ONT resolve the events when they occur.
    • ISP has since replaced ONT.
    • Config was imported from the 7010, omitting any package config.
    • Have tried 3 different Nics for the Wan IF
    • LAN IF is using the onboard Broadcom Nic
    • Am not positive on the exact version of PFSense that was on the 7010, I had selected the stable branch and was using whatever it said was up to date.

    Could there perhaps be something config related that got corrupted on import and is causing the issues?

    ------------- TraceRt from router WAN IF -------------------

     1  * * *
     2  ptld-agw1.inet.qwest.net (207.225.86.145)  1878.017 ms * *
     3  * * *
     4  63-158-222-114.dia.static.qwest.net (63.158.222.114)  1454.335 ms  260.238 ms  249.101 ms
     5  74.125.243.177 (74.125.243.177)  158.250 ms  342.457 ms
        108.170.245.113 (108.170.245.113)  1406.735 ms
     6  * * *
     7  * * dns.google (8.8.8.8)  1637.087 ms
    

    ------------- Ping from router Wan IF ------------------------

    PING 8.8.8.8 (8.8.8.8) from 71.36.127.88: 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=158.006 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=544.022 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=1948.327 ms
    
    --- 8.8.8.8 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 158.006/883.452/1948.327/769.295 ms
    

    ------------- TraceRt from router Client IF -------------------

    1  ptld-dsl-gw51.ptld.qwest.net (207.225.84.51)  49.551 ms  356.669 ms  1215.833 ms
     2  ptld-agw1.inet.qwest.net (207.225.86.145)  443.809 ms  1596.672 ms  1844.559 ms
     3  * sea-edge-12.inet.qwest.net (67.14.41.58)  1581.644 ms  14.294 ms
     4  63-158-222-114.dia.static.qwest.net (63.158.222.114)  22.815 ms  8.851 ms  8.167 ms
     5  74.125.243.177 (74.125.243.177)  14.913 ms
        108.170.245.97 (108.170.245.97)  8.941 ms
        74.125.243.193 (74.125.243.193)  26.185 ms
     6  74.125.253.67 (74.125.253.67)  169.668 ms
        108.170.233.153 (108.170.233.153)  1183.524 ms
        209.85.254.247 (209.85.254.247)  1935.290 ms
     7  * * *
     8  * * *
     9  * * *
    10  * * *
    11  * * *
    12  * * *
    13  * * *
    14  * * *
    15  * * *
    16  * * *
    17  * * *
    18  * * *
    

    ------------- Ping from router Client IF -----------------------

    PING 8.8.8.8 (8.8.8.8) from 192.168.3.1: 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=1845.914 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=2216.709 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=3239.383 ms
    
    --- 8.8.8.8 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 1845.914/2434.002/3239.383/589.266 ms
    

    ----------------- Info from Status > Gateways -------------------

    WAN01_CENTURYLINK_PPPOE (default)	207.225.84.51	8.8.4.4	1210.212ms	799.825ms	0.0%	Offline	Interface WAN01_CENTURYLINK_PPPOE Gateway
    

    -------------------- System Logs ---------------------------
    (I tried disconnecting and reconnecting around 11:18 at which point it begins to throw Unexpected Protocol IP, Could this hint towards the issue?)

    Aug 31 09:58:06	check_reload_status		Syncing firewall
    Aug 31 11:03:33	rc.gateway_alarm	87218	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:534.974ms RTTsd:880.397ms Loss:1%)
    Aug 31 11:03:33	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:03:33	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:03:33	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:03:33	check_reload_status		Reloading filter
    Aug 31 11:03:34	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:03:34	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:05:53	rc.gateway_alarm	59267	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4196.251ms RTTsd:1499.645ms Loss:21%)
    Aug 31 11:05:53	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:05:53	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:05:53	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:05:53	check_reload_status		Reloading filter
    Aug 31 11:05:54	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:05:55	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:07:44	php-fpm	73087	/index.php: Successful login for user 'admin' from: 192.168.3.157 (Local Database)
    Aug 31 11:07:45	rc.gateway_alarm	33853	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3838.708ms RTTsd:1985.755ms Loss:11%)
    Aug 31 11:07:45	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:07:45	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:07:45	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:07:45	check_reload_status		Reloading filter
    Aug 31 11:07:46	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:07:46	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:10:19	rc.gateway_alarm	69490	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:3395.401ms RTTsd:1821.221ms Loss:21%)
    Aug 31 11:10:19	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:10:19	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:10:19	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:10:19	check_reload_status		Reloading filter
    Aug 31 11:10:20	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:10:20	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:10:29	rc.gateway_alarm	20292	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4367.359ms RTTsd:1701.643ms Loss:18%)
    Aug 31 11:10:29	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:10:29	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:10:29	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:10:29	check_reload_status		Reloading filter
    Aug 31 11:10:30	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:10:31	php-fpm	346	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:10:32	rc.gateway_alarm	72163	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4591.740ms RTTsd:1589.594ms Loss:21%)
    Aug 31 11:10:32	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:10:32	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:10:32	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:10:32	check_reload_status		Reloading filter
    Aug 31 11:10:33	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:10:34	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:11:01	rc.gateway_alarm	74351	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4430.263ms RTTsd:2115.223ms Loss:16%)
    Aug 31 11:11:01	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:11:01	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:11:01	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:11:01	check_reload_status		Reloading filter
    Aug 31 11:11:02	php-fpm	346	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:11:02	php-fpm	73087	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:18:08	ppp		caught fatal signal TERM
    Aug 31 11:18:08	ppp		[wan] IFACE: Close event
    Aug 31 11:18:08	ppp		[wan] IPCP: Close event
    Aug 31 11:18:08	ppp		[wan] IPCP: state change Opened --> Closing
    Aug 31 11:18:08	ppp		[wan] IPCP: SendTerminateReq #4
    Aug 31 11:18:08	ppp		[wan] IPCP: LayerDown
    Aug 31 11:18:08	check_reload_status		Rewriting resolv.conf
    Aug 31 11:18:08	ppp		[wan] IFACE: Down event
    Aug 31 11:18:08	ppp		[wan] IFACE: Rename interface pppoe0 to pppoe0
    Aug 31 11:18:08	ppp		[wan] IPV6CP: Close event
    Aug 31 11:18:08	ppp		[wan] IPV6CP: state change Stopped --> Closed
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:08	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:09	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan] IPCP: SendTerminateReq #5
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
    Aug 31 11:18:10	ppp		[wan_link0] rec'd unexpected protocol IP
           **{{{{{{{{{{I deleted 60 or so more repeats of the unexpected Protocol IP error due to character limits in post.}}}}}}}}}}}}}}}**
    Aug 31 11:18:10	ppp		[wan] Bundle: Shutdown
    Aug 31 11:18:10	ppp		[wan_link0] Link: Shutdown
    Aug 31 11:18:10	ppp		process 26141 terminated
    Aug 31 11:18:13	ppp		Multi-link PPP daemon for FreeBSD
    Aug 31 11:18:13	ppp		process 9794 started, version 5.8 (root@pfSense_v2_4_5_amd64-pfSense_v2_4_5-job-04 20:28 17-Dec-2019)
    Aug 31 11:18:13	ppp		web: web is not running
    Aug 31 11:18:13	ppp		[wan] Bundle: Interface ng0 created
    Aug 31 11:18:13	ppp		[wan_link0] Link: OPEN event
    Aug 31 11:18:13	kernel		ng0: changing name to 'pppoe0'
    Aug 31 11:18:13	ppp		[wan_link0] LCP: Open event
    Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Initial --> Starting
    Aug 31 11:18:13	ppp		[wan_link0] LCP: LayerStart
    Aug 31 11:18:13	ppp		[wan_link0] PPPoE: Connecting to ''
    Aug 31 11:18:13	ppp		PPPoE: rec'd ACNAME "ptld-dsl-gw51.ptld.qwest.net"
    Aug 31 11:18:13	ppp		[wan_link0] PPPoE: connection successful
    Aug 31 11:18:13	ppp		[wan_link0] Link: UP event
    Aug 31 11:18:13	ppp		[wan_link0] LCP: Up event
    Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Starting --> Req-Sent
    Aug 31 11:18:13	ppp		[wan_link0] LCP: SendConfigReq #1
    Aug 31 11:18:13	ppp		[wan_link0] PROTOCOMP
    Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
    Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x2004df36
    Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Configure Request #9 (Req-Sent)
    Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
    Aug 31 11:18:13	ppp		[wan_link0] AUTHPROTO CHAP MD5
    Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x08202657
    Aug 31 11:18:13	ppp		[wan_link0] LCP: SendConfigAck #9
    Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
    Aug 31 11:18:13	ppp		[wan_link0] AUTHPROTO CHAP MD5
    Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x08202657
    Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Req-Sent --> Ack-Sent
    Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Configure Ack #1 (Ack-Sent)
    Aug 31 11:18:13	ppp		[wan_link0] PROTOCOMP
    Aug 31 11:18:13	ppp		[wan_link0] MRU 1492
    Aug 31 11:18:13	ppp		[wan_link0] MAGICNUM 0x2004df36
    Aug 31 11:18:13	ppp		[wan_link0] LCP: state change Ack-Sent --> Opened
    Aug 31 11:18:13	ppp		[wan_link0] LCP: auth: peer wants CHAP, I want nothing
    Aug 31 11:18:13	ppp		[wan_link0] LCP: LayerUp
    Aug 31 11:18:13	ppp		[wan_link0] CHAP: rec'd CHALLENGE #244 len: 59
    Aug 31 11:18:13	ppp		[wan_link0] Name: "JUNOS"
    Aug 31 11:18:13	ppp		[wan_link0] CHAP: Using authname "myerswilliam488@centurylink.net"
    Aug 31 11:18:13	ppp		[wan_link0] CHAP: sending RESPONSE #244 len: 52
    Aug 31 11:18:13	ppp		[wan_link0] CHAP: rec'd SUCCESS #244 len: 4
    Aug 31 11:18:13	ppp		[wan_link0] LCP: authorization successful
    Aug 31 11:18:13	ppp		[wan_link0] Link: Matched action 'bundle "wan" ""'
    Aug 31 11:18:13	ppp		[wan_link0] Link: Join bundle "wan"
    Aug 31 11:18:13	ppp		[wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
    Aug 31 11:18:13	ppp		[wan] IPCP: Open event
    Aug 31 11:18:13	ppp		[wan] IPCP: state change Initial --> Starting
    Aug 31 11:18:13	ppp		[wan] IPCP: LayerStart
    Aug 31 11:18:13	ppp		[wan] IPV6CP: Open event
    Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Initial --> Starting
    Aug 31 11:18:13	ppp		[wan] IPV6CP: LayerStart
    Aug 31 11:18:13	ppp		[wan] IPCP: Up event
    Aug 31 11:18:13	ppp		[wan] IPCP: state change Starting --> Req-Sent
    Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #1
    Aug 31 11:18:13	ppp		[wan] IPADDR 0.0.0.0
    Aug 31 11:18:13	ppp		[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Aug 31 11:18:13	ppp		[wan] IPV6CP: Up event
    Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Starting --> Req-Sent
    Aug 31 11:18:13	ppp		[wan] IPV6CP: SendConfigReq #1
    Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Request #248 (Req-Sent)
    Aug 31 11:18:13	ppp		[wan] IPADDR 207.225.84.51
    Aug 31 11:18:13	ppp		[wan] 207.225.84.51 is OK
    Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigAck #248
    Aug 31 11:18:13	ppp		[wan] IPADDR 207.225.84.51
    Aug 31 11:18:13	ppp		[wan] IPCP: state change Req-Sent --> Ack-Sent
    Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
    Aug 31 11:18:13	ppp		[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
    Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #2
    Aug 31 11:18:13	ppp		[wan] IPADDR 0.0.0.0
    Aug 31 11:18:13	ppp		[wan_link0] LCP: rec'd Protocol Reject #10 (Opened)
    Aug 31 11:18:13	ppp		[wan_link0] LCP: protocol IPV6CP was rejected
    Aug 31 11:18:13	ppp		[wan] IPV6CP: protocol was rejected by peer
    Aug 31 11:18:13	ppp		[wan] IPV6CP: state change Req-Sent --> Stopped
    Aug 31 11:18:13	ppp		[wan] IPV6CP: LayerFinish
    Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
    Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
    Aug 31 11:18:13	ppp		[wan] 71.36.127.88 is OK
    Aug 31 11:18:13	ppp		[wan] IPCP: SendConfigReq #3
    Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
    Aug 31 11:18:13	ppp		[wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
    Aug 31 11:18:13	ppp		[wan] IPADDR 71.36.127.88
    Aug 31 11:18:13	ppp		[wan] IPCP: state change Ack-Sent --> Opened
    Aug 31 11:18:13	ppp		[wan] IPCP: LayerUp
    Aug 31 11:18:13	ppp		[wan] 71.36.127.88 -> 207.225.84.51
    Aug 31 11:18:14	check_reload_status		rc.newwanip starting pppoe0
    Aug 31 11:18:14	ppp		[wan] IFACE: Up event
    Aug 31 11:18:14	ppp		[wan] IFACE: Rename interface ng0 to pppoe0
    Aug 31 11:18:14	rc.gateway_alarm	11603	>>> Gateway alarm: WAN01_CENTURYLINK_PPPOE (Addr:8.8.4.4 Alarm:1 RTT:4764.745ms RTTsd:1320.248ms Loss:21%)
    Aug 31 11:18:14	check_reload_status		updating dyndns WAN01_CENTURYLINK_PPPOE
    Aug 31 11:18:14	check_reload_status		Restarting ipsec tunnels
    Aug 31 11:18:14	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Aug 31 11:18:14	check_reload_status		Reloading filter
    Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.
    Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: rc.newwanip: on (IP address: 71.36.127.88) (interface: WAN01_CENTURYLINK[wan]) (real interface: pppoe0).
    Aug 31 11:18:15	dhcpleases		/etc/hosts changed size from original!
    Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 207.225.84.51
    Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: Default gateway setting Interface WAN01_CENTURYLINK_PPPOE Gateway as default.
    Aug 31 11:18:15	php-fpm	73087	/rc.newwanip: IP Address has changed, killing states on former IP Address 71.36.112.131.
    Aug 31 11:18:16	php-fpm	347	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN01_CENTURYLINK_PPPOE.
    Aug 31 11:18:17	dhcpleases		/etc/hosts changed size from original!
    Aug 31 11:18:17	dhcpleases		Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Aug 31 11:18:20	dhcpleases		kqueue error: unknown
    Aug 31 11:18:22	php-fpm	346	/rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wancustom''0.cache: 71.36.127.88
    Aug 31 11:18:22	php-fpm	346	/rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully!
    Aug 31 11:18:22	php-fpm	73087	/rc.newwanip: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Aug 31 11:18:23	php-fpm	73087	/rc.newwanip: Resyncing OpenVPN instances for interface WAN01_CENTURYLINK.
    Aug 31 11:18:23	php-fpm	73087	OpenVPN terminate old pid: 64959
    Aug 31 11:18:23	kernel		ovpns1: link state changed to DOWN
    Aug 31 11:18:23	check_reload_status		Reloading filter
    Aug 31 11:18:23	kernel		ovpns1: link state changed to UP
    Aug 31 11:18:23	php-fpm	73087	OpenVPN PID written: 98835
    Aug 31 11:18:23	check_reload_status		Reloading filter
    Aug 31 11:18:23	check_reload_status		rc.newwanip starting ovpns1
    Aug 31 11:18:23	php-fpm	73087	OpenVPN terminate old pid: 91710
    Aug 31 11:18:23	kernel		ovpns3: link state changed to DOWN
    Aug 31 11:18:24	kernel		ovpns3: link state changed to UP
    Aug 31 11:18:24	php-fpm	73087	OpenVPN PID written: 20898
    Aug 31 11:18:24	php-fpm	73087	/rc.newwanip: Creating rrd update script
    Aug 31 11:18:24	check_reload_status		rc.newwanip starting ovpns3
    Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip: Info: starting on ovpns1.
    Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip: on (IP address: 192.168.31.1) (interface: []) (real interface: ovpns1).
    Aug 31 11:18:24	php-fpm	346	/rc.newwanip: rc.newwanip called with empty interface.
    Aug 31 11:18:24	check_reload_status		Reloading filter
    Aug 31 11:18:24	php-fpm	346	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.31.1 - Restarting packages.
    Aug 31 11:18:24	check_reload_status		Starting packages
    Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip: Info: starting on ovpns3.
    Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip: on (IP address: 192.168.32.1) (interface: []) (real interface: ovpns3).
    Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: rc.newwanip called with empty interface.
    Aug 31 11:18:25	check_reload_status		Reloading filter
    Aug 31 11:18:25	php-fpm	86289	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 192.168.32.1 - Restarting packages.
    Aug 31 11:18:25	check_reload_status		Starting packages
    Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Restarting/Starting all packages.
    Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Stopping service avahi
    Aug 31 11:18:25	avahi-daemon	71257	Got SIGTERM, quitting.
    Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1.
    Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1.
    Aug 31 11:18:25	avahi-daemon	71257	Leaving mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1.
    Aug 31 11:18:25	avahi-daemon	71257	avahi-daemon 0.7 exiting.
    Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Starting service avahi
    Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Stopping service nut
    Aug 31 11:18:25	upsmon	16972	Signal 15: exiting
    Aug 31 11:18:25	upsd	17558	User local-monitor@::1 logged out from UPS [TrippLite_SMART1500LCD]
    Aug 31 11:18:25	upsd	17558	mainloop: Interrupted system call
    Aug 31 11:18:25	upsd	17558	Signal 15: exiting
    Aug 31 11:18:25	usbhid-ups	17176	Signal 15: exiting
    Aug 31 11:18:25	php-fpm	346	/rc.start_packages: Starting service nut
    Aug 31 11:18:25	upsmon	78411	Startup successful
    Aug 31 11:18:25	usbhid-ups	79004	Startup successful
    Aug 31 11:18:25	avahi-daemon	75938	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
    Aug 31 11:18:25	avahi-daemon	75938	Successfully dropped root privileges.
    Aug 31 11:18:25	avahi-daemon	75938	avahi-daemon 0.7 starting up.
    Aug 31 11:18:25	avahi-daemon	75938	WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
    Aug 31 11:18:25	avahi-daemon	75938	Loading service file /usr/local/etc/avahi/services/sftp-ssh.service.
    Aug 31 11:18:25	avahi-daemon	75938	Loading service file /usr/local/etc/avahi/services/ssh.service.
    Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.4.IPv4 with address 192.168.5.1.
    Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.4.IPv4 for mDNS.
    Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.3.IPv4 with address 192.168.4.1.
    Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.3.IPv4 for mDNS.
    Aug 31 11:18:25	avahi-daemon	75938	Joining mDNS multicast group on interface bce0.2.IPv4 with address 192.168.3.1.
    Aug 31 11:18:25	avahi-daemon	75938	New relevant interface bce0.2.IPv4 for mDNS.
    Aug 31 11:18:25	avahi-daemon	75938	Network interface enumeration completed.
    Aug 31 11:18:25	avahi-daemon	75938	Server startup complete. Host name is Camelot.local. Local service cookie is 1381888320.
    Aug 31 11:18:25	avahi-daemon	75938	Failed to add service 'Camelot' of type '_ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/ssh.service): Not permitted
    Aug 31 11:18:25	avahi-daemon	75938	Failed to add service 'Camelot' of type '_sftp-ssh._tcp', ignoring service group (/usr/local/etc/avahi/services/sftp-ssh.service): Not permitted
    Aug 31 11:18:25	avahi-daemon	75027	Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
    Aug 31 11:18:25	avahi-daemon	75027	Successfully dropped root privileges.
    Aug 31 11:18:25	avahi-daemon	75027	open(/var/run/avahi-daemon//pid): File exists
    Aug 31 11:18:25	avahi-daemon	75027	Failed to create PID file: File exists
    Aug 31 11:18:26	php-fpm	73087	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 71.36.112.131 -> 71.36.127.88 - Restarting packages.
    

  • Netgate Administrator

    Most of that taken when it was down?

    Was something rebooted at some point in that log? When?



  • @stephenw10
    The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.

    About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface

    Aug 31 11:18:08 ppp caught fatal signal TERM

    I didn't reboot until ~11:28 or so.

    This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
    Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.

    Thoughts?

    Here are my nuclear options if I can't figure out anything else.

    • Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
    • Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
    • Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
    • Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).


  • I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC


  • Netgate Administrator

    Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

    You are running 2.4.5p1 right?

    Steve



  • @stephenw10 said in WAN interface stops working every few days.:

    Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

    You are running 2.4.5p1 right?

    Steve
    Yep, 2.4.5 -p1
    I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...

    No Mac Spoofing

    System pfSense
    Netgate Device ID: ff022c73b01fa88921e4
    BIOS Vendor: Dell Inc.
    Version: 2.10.0
    Release Date: Thu May 24 2018
    Version 2.4.5-RELEASE-p1 (amd64)
    built on Tue Jun 02 17:51:17 EDT 2020
    FreeBSD 11.3-STABLE

    The system is on the latest version.
    Version information updated at Mon Aug 31 15:14:55 PDT 2020
    CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
    Current: 2300 MHz, Max: 2301 MHz
    4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
    AES-NI CPU Crypto: Yes (active)
    Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
    Kernel PTI Enabled
    MDS Mitigation Inactive


  • Netgate Administrator

    Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.

    2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.

    Steve



  • @gawainxx said in WAN interface stops working every few days.:

    • Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?

    If your network setup isn't too complicated, this is what I would have done by now.

    If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.

    Jeff



  • @akuma1x
    What sort of hardware issues do you think could potentially cause this behavior?

    I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?

    Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head

    .P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_

    If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)



  • @gawainxx

    Could a NAT rule for a Nintendo switch cause any issues?

    	<outbound>
    		<mode>hybrid</mode>
    		<rule>
    			<source>
    				<network>192.168.3.30/32</network>
    			</source>
    			<sourceport></sourceport>
    			<descr><![CDATA[Nindento Switch|Static NAT]]></descr>
    			<target></target>
    			<targetip></targetip>
    			<targetip_subnet></targetip_subnet>
    			<interface>wan</interface>
    			<poolopts></poolopts>
    			<source_hash_key></source_hash_key>
    			<staticnatport></staticnatport>
    			<destination>
    				<any></any>
    			</destination>
    			<updated>
    				<time>1589685349</time>
    				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
    			</updated>
    			<created>
    				<time>1589685349</time>
    				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
    			</created>
    		</rule>
    

    I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm


  • Netgate Administrator

    No, an outbound NAT rule will not be doing anything.

    Traffic shaping is far more likely. Assuming it's anything config related.

    Steve



  • Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.

    Problem still persists.

    There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.

    Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?

    I'm getting towards the end of my list of ideas and could desperately use some solutions.

    I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?

    Here is a copy of my config, I have scrubbed anything cert or credential related from it.
    1599534090821-config_scrubbed.xml

    I'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)


  • Netgate Administrator

    The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.

    If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.

    Steve



  • What version of pfsense??


  • Netgate Administrator

    It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5. 😉



  • @stephenw10 said in WAN interface stops working every few days.:

    The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.

    If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.

    Steve

    I'm not using the ISP router for routing or dhcp atm, just handling the vlan tagged traffic to see if it has any influence...
    I may have to suffer and try running a double NAT for a week or two though to see if the behaviour persists when ISP router handles traffic.



  • @stephenw10 said in WAN interface stops working every few days.:

    It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5. 😉

    Interesting, I'll need to take a close look at that thread later. The webui does definately take several seconds to load when I initially try to access it while the gateway issues are occuring


  • Netgate Administrator

    If you are somehow hitting that still you would see high latency to the firewall itself from a LAN side client everytime you ran Status > Filter reload.

    Steve


Log in to reply