SG-2440 Upload Speed Limited After a Few Minuites
-
@stephenw10 Yeah, it's a pretty unique setup that Comcast has. I have tested without anything connected to the 10.1.10.x subnet as that's how I usually operate. I really only connect a laptop for troubleshooting, etc. I have not tested the laptop on x.x.x.185 with another laptop also on 10.1.10.x. I should be able to try this out though. (Probably tomorrow.) This is a good idea that I didn't think of.
@mer Yes, the router and switch are part of the modem. Basically anything in the gray box are part of the modem. I just drew out the router and switch to show what I think is logically inside the modem box. The modem basically has 4 Ethernet ports on the back that are all equivalent. I can use my static IP or the 10.1.10.x network from any of them. (Note: I have tried using different ports as I wanted to rule out a bad port on the modem.)
@bPsdTZpW I can do this but there is no DHCP setup on the WAN of the pfSense. It's statically configured as the x.x.x.185 IP with a gateway of x.x.x.186. If I were to change the pfSense WAN to DHCP, then it would get the same IP range as the laptops do... 10.1.10.x with a gateway of 10.1.10.1. (Note: If you browse the internet from a host with a 10.1.10.x IP, an external server will see the source as coming from the x.x.x.186 IP.)
With the pfSense WAN being setup statically, the default gateway is actually different than the laptops that are DHCP. But, it is not different when I set up the laptop statically with the same IP and gateway as the pfSense. (Note: when I do this I do NOT also connect the pfSense.) The really weird thing to me is that in this case (the laptop having the static IP of x.x.x.185 with gateway of x.x.x.186) it works and I can get to the web page of the modem.
Good call on the possibility of my modem config being accessed remotely. I didn't think about that before, but good thing... I just checked and it is also not accessible remotely. If it's not the TTL, I'm not sure how the modem knows the difference and ignores the pfSense but not the laptop.
@Patch I do have all of my network behind the pfSense and the pfSense is directly connected to the modem. The image is just showing what the logical implementation of the modem is and what the IPs are. The modem is everything contained in the gray box including the built in 4-port switch. Hopefully that clears up what I was trying to show.
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
If it's not the TTL, I'm not sure how the modem knows the difference and ignores the pfSense but not the laptop.
A pcap started just before connecting pfSense to the modem might give us clues on that. I wonder whether the modem is juggling MAC addresses in some weird way, such that it presents x.x.x.186 on MAC m0:m1:m2:m3:m4:m5 to the laptop, but MAC m6:m7:m8:m9:m10:m11 to pfSense; pfSense probably responds to some ARP packets to which the laptop does not.
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
I do have all of my network behind the pfSense and the pfSense is directly connected to the modem. The image is just showing what the logical implementation of the modem is and what the IPs are. The modem is everything contained in the gray box including the built in 4-port switch.
-
I'm recommending you only connect pfsense to the internal switch in the Comcast modem/router (ie approximate as well as you can putting the Comcast modem/router in bridge mode).
-
Laptops currently with IP address 10.1.10.50 & 10.1.10.51 should be on the Lan not Wan side of pfsense.
-
Doing so simplifies network management.
-
-
@bPsdTZpW This is a great idea. I currently have a gigabit tap on order that I'd like to put inline to do some extended pcaps. (Why not use this as an excuse to buy a new tool...
) I'll post results when it comes in.@patch said in SG-2440 Upload Speed Limited After a Few Minuites:
I'm recommending you only connect pfsense to the internal switch in the Comcast modem/router (ie approximate as well as you can putting the Comcast modem/router in bridge mode).
This is what I'm already doing and have always been set up with.
Laptops currently with IP address 10.1.10.50 & 10.1.10.51 should be on the Lan not Wan side of pfsense.
The laptops in my diagram are only there to show what the IPs are when plugged into the modem. I do not have laptops normally plugged in. I would only plug them in for testing or to get to the modem web config (which for some reason doesn't work from the pfSense LAN... see above.)
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
@bPsdTZpW This is a great idea. I currently have a gigabit tap on order that I'd like to put inline to do some extended pcaps. (Why not use this as an excuse to buy a new tool...
) I'll post results when it comes in.I'm looking forward to this data.
-
@steve1515 I skimmed through the thread. At any point did you power off the Comcast router/modem? Apologies if I missed that. Another long shot is to ask Comcast if they have any security features on their device they can disable. I ask because a few years back we had a small flurry of issues over several months with specific inbound connections being blocked and (usually) restarting or (once) powering off the Comcast router let the IP connect again. I seem to recall being told the mysterious not-documented (as I recall?) security could be turned off, in lieu of rebooting the router.
Does it also happen if you give the pfSense a 10.1.10.x IP?
And yes for everyone else Comcast does also provide 10.1.10.x NAT when "bridged"...they have for at least the 10-15 years we've worked with them. Not necessary of course but is actually useful if you plug a PC into it to test, which is presumably for them to test "around" your router.
-
Just providing an quick update.
It seems that my inline tap has been lost in shipping.
I'm hoping this is temporary and it will get here soon.I did notice something strange though... It seems that the last couple of times that I've unplugged the pfSense WAN cable and plug it back in, the upload speed no longer gets reset to the full 20Mbps. I now seem to have to reboot the pfSense to get the speed back. I haven't made any changes since the previous messages, so I'm not sure why this could be. I guess this might be another clue in this puzzle.
-
I just did another test which showed an interesting result.
I had the SG-1100 plugged in with my usual pfSense config (static IP and all as above), but this time I only connected one device on the LAN side. Everything else was not connected. At 58 past the hour, the upload remained.
This is now looking more like I have some kind of device on my LAN that triggers something in pfSense to limit upload speed. Seems strange...
Any tips on finding this device other than unhooking things each hour?
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
Any tips on finding this device other than unhooking things each hour?
Bisecting search.
- Unhook half of your devices & test
- Then unhook half the devices in the error half & test
- etc
-
Wanted to share an updated on this...
I've continued troubleshooting and I think I've narrowed down the cause although, I'm not sure of the fix. It seems to be caused by a raspberry pi that I use to upload an audio stream. It's a continuous police/fire scanner that ranges anywhere from 25 kbps to 500kbps. If I unplug the box and reboot pfSense, my upload stays at 20mbps.
In trying to prove that it wasn't pfSense, I added an alias IP to the WAN port in the 10.1.10.x network and setup outbound NAT for that network range to use that IP. This was so I could connect a laptop to the "Comcast NAT LAN" (See my image in the posts above.) and run an iperf speed test to a host on my pfSense LAN (by connecting to my WAN's static IP). Doing this shows 600-900 mpbs depending on direction of data flow. This had me thinking that it wasn't the pfSense that was limiting the speed.
Next, I had Comcast take a look and see what they saw... The issue is they don't see anything on their end. The biggest reason, is because when my pfSense LAN is limited to 10mbps upload, you can always plug a laptop in to the 10.1.10.x network on the modem and get 20mbps upload. This, to them, proves it's not Comcast.
I'm not really sure now what the fix would be or if there's anything I can do. Would a continuous upload stream audio stream break pfSense in some way at :58 past every hour? I'm still leaning towards this being a Comcast issue, but I don't really have a way to prove it.
-
Audio stream data like that can be unusual. For example it's not uncommon to see a lot of VoIP traffic swamp a firewall when the total bandwidth doesn't appear to be that high. That's because that sort of traffic is often very small packets and you end up hitting the PPS limits of the firewall at relatively low total bandwidth. Check the traffic graphs for PPS throughput.
Steve
-
@stephenw10 Where would I check on PPS throughput? Under Status->Traffic Graphs is only shows bandwidth in bits/sec.
-
Here:
-
Does this look unusual? In the picture, I'm hovering over :58 past to show more detail at that time.
-
No that's nothing, you'd need to see many thousands of PPS for it to be an issue.
Must be something the Pi is doing then. Check it's logs. Try setting a logging rule to pass only it;s traffic and see what connections it's opening at that time.
Stevce
-
I just did a new test and have an interesting result...
(Note: This is on 22.01)I put the Pi which is uploading on the Comcast 10.1.10.x network and rebooted the pfSense box to get my upload on the LAN (192.168.1.x) backup to 20mpbs.
I figured since the Pi was no longer behind the pfSense that the speed wouldn't drop... what actually happened was my LAN speed dropped as usual to 10mpbs and doing an iPerf test to an outside server showed that the Pi still had an upload of 20mbps.
This is kind of a strange result to me. I'm not really sure what to make of it.
I also did a pcap with an inline sniffer last night on the Pi and saw nothing unusual at :58 past the hour when the speed dropped.
-
So possibly the previous result when removing the Pi seemed to remove the issue was just coincidence?
-
@stephenw10 I suppose. I did it multiple times to verify it did not happen with the Pi disconnected. I'll need to go back and try to eliminate devices again.
The only thing that I can count on is when I have a device disconnected and the upload drops, then it's not the disconnected device.
It's pretty disruptive to try it out, but I might just put the Pi as the single device on with my static IP and see what happens. That would at least eliminate pfSense.
I'm leaning toward this being a Comcast issue, but have no good way to show that if a tech were to come here.
-
Mmm, I agree it's a tricky issue to prove. I can't recall ever seeing that before either.
-
Another interesting test I did today...
I plugged only the Pi and a laptop into the pfSense LAN. I rebooted the pfSense to get back to 20Mbps and waited till :58 past the hour.
The laptop is the same one that is normally not plugged into the network that I was using for tests before which never caused the issue by itself.
So, now we have a test with only those 2 devices plugged into the LAN port. At :58, the upload speed dropped to 10Mbps again. This to me, proves it's the Pi causing the issue.
The really weird thing is that in the last test, it still caused the issue to the pfSense LAN when the Pi was connected to the Comcast 10.x network.
It's almost as if the Pi uploading on either the 10.x or my static IP (pfSense) will cause the static IP to drop to 10Mbps upload. This is very strange and I'm not sure why this would happen.
I'm heavily leaning toward this being a Comcast issue, but still have not good way to prove it.
