Extremely Frustrating Outages
-
@jknott said in Extremely Frustrating Outages:
Please post it here, as others may be able to help.
There might be something useful here or here.In addition to the comments in the first link, you might try reducing MTU on the source computer, in case the packets are being fragmented, but not recovered properly.
I've sent you the link to the file capture. If anyone else would like the file I'd be happy to send it. Over the years I've been here I've seen time and again calls for people to edit out their WAN IP, especially if it's static. I'd hate to just put it out there, along with whatever else the capture may show due to potential security concerns. I'm not adverse to sharing it with others who are willing to help, though.
-
@stephenw10 Which is exactly the conclusion we kept coming to except all the power and signals look good on the modem and at the first site, all the issues went away once we removed the devices from the wireless. At the first site it's 1 of 3 laptops, just not sure which at the moment. All of my diagnostics (at both sites) points to an issue with the modem. What Spectrum is saying is that it is one of our devices causing it to have issues because there is nothing in the logs. (Of course, there's never anything in the logs and one of the first things they do is reboot the modem which clears out all the counters anyway.)
At the second company there is a second modem for the security cameras that is plugged into the same splitterthat I am also pinging which has no issues. Spectrum has replaced the data modem, cable, and splitter and the issue persists.
-
Hmm, yeah tough to diagnose. But ig you still see packet loss and latency with a laptop connected to the modem it can't really be anything else IMO.
Steve
-
I haven't gone through a lot of that yet, but I'm not seeing many errors. In fact, in about a thousand packets, there has only been 1 dup ack and no other errors at all. However, I'm seeing a lot of DNS traffic.
-
What I was posting you can start seeing around packet 1190. But yes, lots of DNS. I don't know how normal it is. In fact, to look through the capture I used !(dns) as a filter.
-
@jknott Have you had a chance to look anymore? Anything you see?
-
The one curious thing I see is about 35 duplicate acks that a device on Spectrum (I assume that's you) to Cloudfront. That would indicate a lost ack at the Cloudfront end. That's about all I see.
-
I went through all of the logs for all of our installs and identified a few other sites that appeared to be experiencing something similar. Going on the theory that maybe we are overloading the modem and the large number of DNS packets seen in the captures, I've switched Ubound from Resolver to Forwarder in a some of the units having issues. Since I have OpenDNS in the General tab I also disabled DNSSEC. After a couple of days I can see a trend where the issues have settled down tremendously. It's been enough that I don't think it's coincidence. Any thoughts?
-
Could be. There are some modems that are known to badly handle a lot of UDP traffic. (Intel PUMA based).
What modems are you using?
Steve
-
What brand and model modem are you using? Is it a "gateway" type modem that the ISP has in their style of "bridge mode"? It makes a difference.
-
At the site I was looking through logs with @JKnott, this is the info provided in the modem:
ARRIS DOCSIS 3.0 / PacketCable 2.0 Touchstone Residential Gateway HW_REV: 8 VENDOR: ARRIS Group, Inc. BOOTR: 4.2.0.45 SW_REV: 9.1.103J6TW2.SIP.PC20.TW MODEL: TG1682GI don't know about the other sites. Also, the initial site I was having problems with (that was fixed by removing the laptops from the wireless) went through several model of modems. I believe it is currently also using a TG1682G. In most cases the modems are configure as RIP w/o NAT. In some instances where there is no Static IP then they are in Bridge mode.
-
@stewart said in Extremely Frustrating Outages:
TG1682G
Thanks Stewart. Yes definitely a Puma model. https://badmodems.com/Forum/app.php/badmodems
If Spectrum has not updated firmware on these models.. then you can blame the modem.
Unless you can get Spectrum to provide a non Puma model you can continue to expect problems. Many issues have been fixed by the firmware fixes but many problems persist.
-
@chpalmer said in Extremely Frustrating Outages:
@stewart said in Extremely Frustrating Outages:
TG1682G
Thanks Stewart. Yes definitely a Puma model. https://badmodems.com/Forum/app.php/badmodems
If Spectrum has not updated firmware on these models.. then you can blame the modem.
Unless you can get Spectrum to provide a non Puma model you can continue to expect problems. Many issues have been fixed by the firmware fixes but many problems persist.
That's really nice of you; thinking that Spectrum will update their modem's firmware! We once fought with them for months, even trying to install our own modems, because they had a firmware that broke SIP. (I think that was the TG1672 model) Happened all over across many of our clients. The only solution we found was to run SIP over SSL so it wouldn't know what the traffic was. Even had clients switch VOIP providers because their existing provider couldn't (or wouldn't) do it over SSL. That was with SIP ALG off, Firewall off, etc. Now they look back and laugh at all the headaches it caused but back then they wouldn't take any responsibility for it.
Can you explain what PUMA means?
Edit: I put in the IP of the modem and it does indeed cause the same high packet loss and latency I've been experiencing. -
If you haven't look over that Badmodems site and familiarize yourself with the issue.
Ive had to change out a bunch of modems for customers around my area due to similar issues.
Get a Broadcom based modem if you can.. Im doubtful though with the static IP's you are using but you have to ask. Im not familiar with Spectrum and how they hand out multiples..
-
Unfortunately, Spectrum only allows you to bring your own modem on a residential account. On a Business account you must use their modems. You now get the lottery of either the TB1682G or a new Spectrum branded 2 box solution.
-
Thanks! Adding that information to my knowledge base.. Luckily I only have one customer on Spectrum in North Carolina and they are residential customers. Motorola MB8600 on their account.
Ill I can say is escalate escalate escalate! Enough input by customers and customers reps finally got Comcast to allow COE on their business accounts as long as static IP's are not involved.
-
@chpalmer Escalating with Spectrum gives me "Call us again on this and we'll bill you for coming out." I've had multiple techs go out to both sites. The techs that go on site say they put the modem in their "SCOPE" system which puts them in Device Watch. That allows the techs to go back and look at history for the unit. BUT, when you call in and talk to a CSR every one of them says that they no longer use that system. Only the techs onsite can setup or see into the SCOPE system now but you can't get a tech onsite without the possibility of them billing for every visit. Even then they just troubleshoot the moment and don't even refer to it unless you make it a point to make them. It's crazy. But still better than every DSL provider and AT&T U-Verse in this area. You pick your poison.
-
Maybe try using DNSoverTLS so you're not sending UDP DNS packets at all.
If that improves things further at least it's another data point.Steve
-
@stephenw10 said in Extremely Frustrating Outages:
Maybe try using DNSoverTLS so you're not sending UDP DNS packets at all.
If that improves things further at least it's another data point.Steve
It's only for forwarding that I can see. Is that accurate?
-
Yes, you can only use with forwarding mode.
As a test though it would be useful just eliminate all DNS UDP packets hitting the modem.
Steve
