Extremely Frustrating Outages
-
-
Mmm, that pcap is pretty ugly though.
No packet loss on the WAN when this happens?
Almost looks like asymmetry. I'd still be looking for something on the wifi providing an alternate route somehow.
Steve
-
@stephenw10 said in Extremely Frustrating Outages:
Mmm, that pcap is pretty ugly though.
No packet loss on the WAN when this happens?
Almost looks like asymmetry. I'd still be looking for something on the wifi providing an alternate route somehow.
Steve
From inside the network I ping:
Switch - No packet loss
LAN IP - No Packet loss
WAN IP - Some packet loss when the logs show services restarting due to the Gateway going up and down.
Gateway IP (modem) - Similar packet loss but also high latency during the issues.From outside the network I ping:
Gateway IP (modem) - Packet Loss and High Ping
WAN IP - Packet Loss and High PingIn the case of the first client I also had a laptop plugged directly into the modem with a spare public IP assigned to it. During the issues I would see:
Gateway IP (modem) - Packet Loss and High Ping
WAN IP - Packet Loss which I believe is due to the interface restarting as the gateway goes up and down.If something on the wifi were causing an alternate route, how could that affect me from being able to ping the modem remotely? It would just mess up the packets inside the network, no?
-
Indeed it would not.
From that description it looks far more like an upstream issue. A failing modem or whatever that is connected to.
Steve
-
@jknott said in Extremely Frustrating Outages:
Please post it here, as others may be able to help.
There might be something useful here or here.In addition to the comments in the first link, you might try reducing MTU on the source computer, in case the packets are being fragmented, but not recovered properly.
I've sent you the link to the file capture. If anyone else would like the file I'd be happy to send it. Over the years I've been here I've seen time and again calls for people to edit out their WAN IP, especially if it's static. I'd hate to just put it out there, along with whatever else the capture may show due to potential security concerns. I'm not adverse to sharing it with others who are willing to help, though.
-
@stephenw10 Which is exactly the conclusion we kept coming to except all the power and signals look good on the modem and at the first site, all the issues went away once we removed the devices from the wireless. At the first site it's 1 of 3 laptops, just not sure which at the moment. All of my diagnostics (at both sites) points to an issue with the modem. What Spectrum is saying is that it is one of our devices causing it to have issues because there is nothing in the logs. (Of course, there's never anything in the logs and one of the first things they do is reboot the modem which clears out all the counters anyway.)
At the second company there is a second modem for the security cameras that is plugged into the same splitterthat I am also pinging which has no issues. Spectrum has replaced the data modem, cable, and splitter and the issue persists.
-
Hmm, yeah tough to diagnose. But ig you still see packet loss and latency with a laptop connected to the modem it can't really be anything else IMO.
Steve
-
I haven't gone through a lot of that yet, but I'm not seeing many errors. In fact, in about a thousand packets, there has only been 1 dup ack and no other errors at all. However, I'm seeing a lot of DNS traffic.
-
What I was posting you can start seeing around packet 1190. But yes, lots of DNS. I don't know how normal it is. In fact, to look through the capture I used !(dns) as a filter.
-
@jknott Have you had a chance to look anymore? Anything you see?
-
The one curious thing I see is about 35 duplicate acks that a device on Spectrum (I assume that's you) to Cloudfront. That would indicate a lost ack at the Cloudfront end. That's about all I see.
-
I went through all of the logs for all of our installs and identified a few other sites that appeared to be experiencing something similar. Going on the theory that maybe we are overloading the modem and the large number of DNS packets seen in the captures, I've switched Ubound from Resolver to Forwarder in a some of the units having issues. Since I have OpenDNS in the General tab I also disabled DNSSEC. After a couple of days I can see a trend where the issues have settled down tremendously. It's been enough that I don't think it's coincidence. Any thoughts?
-
Could be. There are some modems that are known to badly handle a lot of UDP traffic. (Intel PUMA based).
What modems are you using?
Steve
-
What brand and model modem are you using? Is it a "gateway" type modem that the ISP has in their style of "bridge mode"? It makes a difference.
-
At the site I was looking through logs with @JKnott, this is the info provided in the modem:
ARRIS DOCSIS 3.0 / PacketCable 2.0 Touchstone Residential Gateway HW_REV: 8 VENDOR: ARRIS Group, Inc. BOOTR: 4.2.0.45 SW_REV: 9.1.103J6TW2.SIP.PC20.TW MODEL: TG1682G
I don't know about the other sites. Also, the initial site I was having problems with (that was fixed by removing the laptops from the wireless) went through several model of modems. I believe it is currently also using a TG1682G. In most cases the modems are configure as RIP w/o NAT. In some instances where there is no Static IP then they are in Bridge mode.
-
@stewart said in Extremely Frustrating Outages:
TG1682G
Thanks Stewart. Yes definitely a Puma model. https://badmodems.com/Forum/app.php/badmodems
If Spectrum has not updated firmware on these models.. then you can blame the modem.
Unless you can get Spectrum to provide a non Puma model you can continue to expect problems. Many issues have been fixed by the firmware fixes but many problems persist.
-
@chpalmer said in Extremely Frustrating Outages:
@stewart said in Extremely Frustrating Outages:
TG1682G
Thanks Stewart. Yes definitely a Puma model. https://badmodems.com/Forum/app.php/badmodems
If Spectrum has not updated firmware on these models.. then you can blame the modem.
Unless you can get Spectrum to provide a non Puma model you can continue to expect problems. Many issues have been fixed by the firmware fixes but many problems persist.
That's really nice of you; thinking that Spectrum will update their modem's firmware! We once fought with them for months, even trying to install our own modems, because they had a firmware that broke SIP. (I think that was the TG1672 model) Happened all over across many of our clients. The only solution we found was to run SIP over SSL so it wouldn't know what the traffic was. Even had clients switch VOIP providers because their existing provider couldn't (or wouldn't) do it over SSL. That was with SIP ALG off, Firewall off, etc. Now they look back and laugh at all the headaches it caused but back then they wouldn't take any responsibility for it.
Can you explain what PUMA means?
Edit: I put in the IP of the modem and it does indeed cause the same high packet loss and latency I've been experiencing. -
If you haven't look over that Badmodems site and familiarize yourself with the issue.
Ive had to change out a bunch of modems for customers around my area due to similar issues.
Get a Broadcom based modem if you can.. Im doubtful though with the static IP's you are using but you have to ask. Im not familiar with Spectrum and how they hand out multiples..
-
Unfortunately, Spectrum only allows you to bring your own modem on a residential account. On a Business account you must use their modems. You now get the lottery of either the TB1682G or a new Spectrum branded 2 box solution.
-
Thanks! Adding that information to my knowledge base.. Luckily I only have one customer on Spectrum in North Carolina and they are residential customers. Motorola MB8600 on their account.
Ill I can say is escalate escalate escalate! Enough input by customers and customers reps finally got Comcast to allow COE on their business accounts as long as static IP's are not involved.