Extremely Frustrating Outages
-
Ok pcap on the WAN when this is happening. Try to access the modem. See what's happening in the capture. Is the modem actually talking that long to respond? Errors? Re-transmissions?
Steve
-
@stephenw10 said in Extremely Frustrating Outages:
Ok pcap on the WAN when this is happening. Try to access the modem. See what's happening in the capture. If the modem actually talking thar long to respond? Errors? Re-transmissions?
Steve
This site is remote to me so I'm a little limited on what I can do. I was remotely connected into the firewall when they started having issues again. I managed to get a pcap but couldn't connect to a PC to try to log into the modem as the service was too bad. By the time I got in the service had corrected itself. I'm not sure what I'm looking for in the pcap, though. Normally I'd go pick it apart by protocol to diagnose an SMB, FTP or SIP issue. I do see a lot of Protocol=QUIC, Info=.....Len=55[Malformed Packet]. Not sure how normal that is.
-
@stewart said in Extremely Frustrating Outages:
QUIC
Here's what QUIC is. If you're getting malformed packets, that tends to indicate a hardware issue nearby. Malformed packets shouldn't be passing through routers or switches, as they'd be caught with the CRC check. What MAC address are they coming from? That would indicate the failing hardware.
-
Hardware offloading in the NIC can make the checksum appear invalid in a pcap.
I would disable all hardware offloading anyway in Sys > Adv > Net.
Steve
-
@jknott said in Extremely Frustrating Outages:
@stewart said in Extremely Frustrating Outages:
QUIC
Here's what QUIC is. If you're getting malformed packets, that tends to indicate a hardware issue nearby. Malformed packets shouldn't be passing through routers or switches, as they'd be caught with the CRC check. What MAC address are they coming from? That would indicate the failing hardware.
I see the Malformed Packets coming into my pfSense box from the modem MAC address but I also see them leaving my pfSense box going into the modem MAC address. That would indicate that Wireshark is saying that packets coming and going are all malformed. Perhaps that is due to the Hardware offloading that @stephenw10 was mentioning?
-
I've now checked the Disable hardware checksum offload box.
I did manage to get another packet capture. There are hundreds, if not more, of
-TCP Retransmissions
-TCP Dup ACK
-TCP Out of Order
-TCP Previous segment not captured -
-
You mean throw a switch in there with port mirroring into a PC and run wireshark on there?
-
Yes, just in case the pfsense NIC is the source. If the errors appear in Packet Capture, but not Wireshark that's likely the cause.
-
@jknott said in Extremely Frustrating Outages:
Yes, just in case the pfsense NIC is the source. If the errors appear in Packet Capture, but not Wireshark that's likely the cause.
In the first site that had this issue, that's what I thought as a possibility so I swapped the firewall. Can't say for sure that it's the same as this site but at the last site it didn't help. The errors persisted across 2 firewalls.
-
Here's a snippet from when things are bad.
-
Can you upload the capture?
-
@jknott I can tomorrow, but wouldn't want it public? How should I send it to you?
-
-
Mmm, that pcap is pretty ugly though.
No packet loss on the WAN when this happens?
Almost looks like asymmetry. I'd still be looking for something on the wifi providing an alternate route somehow.
Steve
-
@stephenw10 said in Extremely Frustrating Outages:
Mmm, that pcap is pretty ugly though.
No packet loss on the WAN when this happens?
Almost looks like asymmetry. I'd still be looking for something on the wifi providing an alternate route somehow.
Steve
From inside the network I ping:
Switch - No packet loss
LAN IP - No Packet loss
WAN IP - Some packet loss when the logs show services restarting due to the Gateway going up and down.
Gateway IP (modem) - Similar packet loss but also high latency during the issues.From outside the network I ping:
Gateway IP (modem) - Packet Loss and High Ping
WAN IP - Packet Loss and High PingIn the case of the first client I also had a laptop plugged directly into the modem with a spare public IP assigned to it. During the issues I would see:
Gateway IP (modem) - Packet Loss and High Ping
WAN IP - Packet Loss which I believe is due to the interface restarting as the gateway goes up and down.If something on the wifi were causing an alternate route, how could that affect me from being able to ping the modem remotely? It would just mess up the packets inside the network, no?
-
Indeed it would not.
From that description it looks far more like an upstream issue. A failing modem or whatever that is connected to.
Steve
-
@jknott said in Extremely Frustrating Outages:
Please post it here, as others may be able to help.
There might be something useful here or here.In addition to the comments in the first link, you might try reducing MTU on the source computer, in case the packets are being fragmented, but not recovered properly.
I've sent you the link to the file capture. If anyone else would like the file I'd be happy to send it. Over the years I've been here I've seen time and again calls for people to edit out their WAN IP, especially if it's static. I'd hate to just put it out there, along with whatever else the capture may show due to potential security concerns. I'm not adverse to sharing it with others who are willing to help, though.
-
@stephenw10 Which is exactly the conclusion we kept coming to except all the power and signals look good on the modem and at the first site, all the issues went away once we removed the devices from the wireless. At the first site it's 1 of 3 laptops, just not sure which at the moment. All of my diagnostics (at both sites) points to an issue with the modem. What Spectrum is saying is that it is one of our devices causing it to have issues because there is nothing in the logs. (Of course, there's never anything in the logs and one of the first things they do is reboot the modem which clears out all the counters anyway.)
At the second company there is a second modem for the security cameras that is plugged into the same splitterthat I am also pinging which has no issues. Spectrum has replaced the data modem, cable, and splitter and the issue persists.
-
Hmm, yeah tough to diagnose. But ig you still see packet loss and latency with a laptop connected to the modem it can't really be anything else IMO.
Steve
