SG-2440 Upload Speed Limited After a Few Minuites
-
I don't see any cron jobs in pfSense that I can see that would cause this. This is also on a new SG-1100 system as well, so I'm not sure what it could be.
Note: I'm not sure if this was clear before, but once the upload is limited it stays limited forever until I either unplug the WAN cable and re-plug it in or I reboot the pfSense box. This happened on both my SG-2440 and the SG-1100 I used for testing.
-
Different DHCP lease times? Like something upstream is expiring the lease and using some default shaping?
Does it matter if 58mins past the hour is 10mins after connecting or nearly 1hour?Anything logged at all at 58mins past?
-
It's a static IP on the WAN side, so I don't think it would be a DHCP issue. It also doesn't seem to matter if I reboot the modem or not or when I connect the WAN cable.
I think why I initially thought it was happening within 15 minutes, was because I was getting lucky and not noticing that it was time related before and was near the top of the hour. It doesn't seem to matter when I plug in though... once it's xx:58 on the clock it seems to happen.
I don't see anything logged at that time either.
-
Hmm, if you disable ntp and set the clock in pfSense differently does it still happen at the same time?
-
Modifying the pfSense clock was a great idea @stephenw10 !
When I disabled NTP and changed the clock on the pfSense box, the pfSense xx:58 time came and went without a loss in upload speed. When the actual time (from another NTP synced clock) hit xx:58 though, I lost my upload speed again.
This seems to go back to it being a modem/Comcast issue. The crazy thing is why only my 2 pfSense boxes even when tried with different MAC's/hostnames and never the 2 laptops?
-
Mmm, indeed, hard to see what it could possibly be. Static WAN IP eliminates most things. Different NIC types.
I guess maybe run a pcap on the WAN at exactly the time is starts clamping and see if it's sending something that pfSense doesn't respond to.
Hard to imagine what that could be that a laptop does respond to though. -
I did a packet capture on the WAN link on the pfSense box and saw nothing unusual. I did the same thing on a laptop just to see if there was anything noticeable... there was not.
Some background on the static IP set up that Comcast has... The modem has a routed /30 subnet where one IP (x.x.x.185) is my static IP and the other is the modem's IP (x.x.x.186). This modem also has the usual 10.x.x.x NAT stuff which I don't use and have disabled all features of including firewall, etc. But if you were to connect more PCs to the modem and use DHCP, you'd get a 10.x.x.x IP and have NAT'd Internet access going out of the x.x.x.186 IP in my static range. It's really "special" how Comcsat does this, and I'd live to be able to just bridge completely, but that's not how it works with Comcast statics. To clearfy some, the pfSense doesn't use DHCP or anything, it's a standard static config. The modem just also still works with NAT if you were to use DHCP. (Note: When testing with the laptops in the posts above, I've tried with the NAT IP and also at different times with my static (x.x.x.185) just like the pfSense. The upload never got cut in half, no matter what I did with the laptops.)
Now that the background info is laid out, one thing that I did notice was that when I was connected to the modem via the laptop with it configured to my static IP of x.x.x.185, I could connect to the modem's IP of x.x.x.186 in a web browser and get to it's config page. This for some reason does not work when I'm behind pfSense on the LAN. From the LAN connecting to x.x.x.186 always times out in the browser and a Wireshark shows that no packets are ever returned from the modem. The weird thing is that it used to work from the LAN a few years ago. One day I noticed it no longer worked from the LAN. When exactly, I don't know as I don't access the modem web page often. I just figured it was Comcast updating their security or something and that was that. I figured I would always just need to plug in a laptop and connect to it's 10.x.x.x gateway address to get to the web page.
Does this possibly indicate something weird happening with my pfSense? Thinking about it... I really don't see why connecting to the x.x.x.186 network from my LAN should fail. The packets should go out on the /30 subnet and the modem should just think it's talking to a local host on a directly attached interface.... just like the laptop could.
I'm kind of shooting in the dark here, but maybe if I toggle the "switch" that makes the web page work something else will start working... Thoughts? Anyone know how to do this?
I guess it's possibly a side question, but any thoughts on why I can't access the web page from the LAN but could from the laptop?
-
A weird addition to the above...
I just did a pcap on the WAN interface while trying to access the modem IP x.x.x.186 from the LAN. I can see that the packets go out from x.x.x.185 (pfSense WAN) to x.x.x.186 (modem IP), but no packets ever come back!
It's almost like the modem is somehow ignoring the pfSense but talks when I use the laptop.
-
@steve1515 : I just did a pcap on the WAN interface while trying to access the modem IP x.x.x.186 from the LAN. I can see that the packets go out from x.x.x.185 (pfSense WAN) to x.x.x.186 (modem IP), but no packets ever come back!
It's almost like the modem is somehow ignoring the pfSense but talks when I use the laptop.
I am surprised that the modem responds to packets containing its WAN address (x.x.x.186) that it receives on its LAN port (when sent via the laptop). Most modems have a LAN-side (private) address for administration. BTW, having a WAN-side address might open the modem to hacking via the WAN.
I wonder whether the modem is routing the packets to x.x.x.186 from pfSense out via its default gateway (that is, onto the WAN), which, if so, is why nothing ever comes back to pfSense.
Does traceroute from pfSense to the speed-test site return different values when your connection is fast and when it's slow?
Also do you happen to have RIP (https://docs.netgate.com/pfsense/en/latest/packages/routed.html ) enabled on pfSense? Or on the modem?
-
Try running a port test to it from pfSense dircetly (Diag > Test Port) that should duplicate what the laptop does exactly.
About the only difference here would be the TTL value of incoming traffic. Packets coming through pfSense have already been routed so would have a lower value. Usually that makes no difference because the TTL is high enough it never gets close to 0.
It's been a while since I looked at it but cell phone providers used to us that as a way on enforcing only a single client when tethering, you could not connect a router to it.Steve
-
@bPsdTZpW I think the modem implementation is a little more like this diagram I drew.
You can see that the modem has both my routed x.x.x.184/30 network with it's interface assigned the x.x.x.186 IP and also the 10.1.10.1 IP for it's NATing. In this diagram the laptops on the 10.1.10.1 network can get to the modems web page by going to 10.1.10.1 and if I were to configure a laptop with the x.x.x.185/30 IP and plug it in place of the pfSense then it could get to the modem web page via the x.x.x.186 IP. The weird thing is when I'm using either the pfSense Port Test or a PC on the pfSense LAN the modem doesn't ever return any packets back to a connection attempt to x.x.x.186 port 80.
Are you saying there's a way to setup static routes to make this work?
I also checked the traceroutes to the test server and there appears to be no change between fast and slow upload times. I also do not have RIP installed on the pfSense. The modem doesn't give me any RIP options in it's config. I also didn't see any RIP related packets when doing pcaps.
@stephenw10 I tried the Port Test and did a pcap while it was happening. I get the same thing... zero packets back from the modem. I took a look at the TTL of the traffic coming out of the pfSense WAN in the captures and it's well above zero. This seems pretty strange that the Port Test would not get any packets back, doesn't it?
This is really starting to look like the modem knows it's a pfSense box and is doing something strange.
-
It's still surprising (to me at least!) that is can do both those things at once.
Did you test it without anything connected locally to the 10.1.10.X subnet?
Or conversely did you test a laptop at the x.x.x.185 IP with another client on the 10.1.10.X subnet?
Steve
-
Are the router and 4port switch all part of the Comcast modem?
Kind of going off of @stephenw10 does the IP follow the port on the switch? -
@steve1515 stephenw10 earlier suggested:
I guess maybe run a pcap on the WAN at exactly the time is starts clamping and see if it's sending something that pfSense doesn't respond to.
I'd suggest a pcap of the entire sequence from plugging (freshly-rebooted) pfSense into the (freshly-rebooted) modem until the speed test begins slowing down. I'm particularly interested in the initial negotiation (e.g. BOOTP/DHCP) sequence between pfSense and the modem. BTW, what gateway does pfSense get from the modem? Is it that same as the gateway that the laptop gets when it's directly connected to the modem?
Are you saying there's a way to setup static routes to make this [connecting from a LAN device to a modem on pfSense's WAN port] work?
The typical way is to have the modem respond to a private (e.g., 192.168.m.n) address, then to set a static route on pfSense to reach that address using pfSense's WAN gateway. Your modem is weird in responding to a WAN address ( x.x.x.186) instead. [1] When you try to reach that address, pfSense sends it out its default gateway. I suspect that the modem gives pfSense a different default gateway (say g1) than it gives the laptop (say g2), and that packets sent out via g1 go directly onto the internet, and nothing ever responds to them, hence you're not seeing anything come back on the pcap. Whereas I suspect g2 is multiplexed internally to the modem, which realizes that x.x.x.186 is its own GUI's address, and routes it accordingly.
[1] This probably means that the modem's GUI can be reached from the internet at large. That would be a security risk.
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
You can see that the modem has
Why do you not have pfsense immediately after your modem/router with all of your network on pfsense
-
@stephenw10 Yeah, it's a pretty unique setup that Comcast has. I have tested without anything connected to the 10.1.10.x subnet as that's how I usually operate. I really only connect a laptop for troubleshooting, etc. I have not tested the laptop on x.x.x.185 with another laptop also on 10.1.10.x. I should be able to try this out though. (Probably tomorrow.) This is a good idea that I didn't think of.
@mer Yes, the router and switch are part of the modem. Basically anything in the gray box are part of the modem. I just drew out the router and switch to show what I think is logically inside the modem box. The modem basically has 4 Ethernet ports on the back that are all equivalent. I can use my static IP or the 10.1.10.x network from any of them. (Note: I have tried using different ports as I wanted to rule out a bad port on the modem.)
@bPsdTZpW I can do this but there is no DHCP setup on the WAN of the pfSense. It's statically configured as the x.x.x.185 IP with a gateway of x.x.x.186. If I were to change the pfSense WAN to DHCP, then it would get the same IP range as the laptops do... 10.1.10.x with a gateway of 10.1.10.1. (Note: If you browse the internet from a host with a 10.1.10.x IP, an external server will see the source as coming from the x.x.x.186 IP.)
With the pfSense WAN being setup statically, the default gateway is actually different than the laptops that are DHCP. But, it is not different when I set up the laptop statically with the same IP and gateway as the pfSense. (Note: when I do this I do NOT also connect the pfSense.) The really weird thing to me is that in this case (the laptop having the static IP of x.x.x.185 with gateway of x.x.x.186) it works and I can get to the web page of the modem.
Good call on the possibility of my modem config being accessed remotely. I didn't think about that before, but good thing... I just checked and it is also not accessible remotely. If it's not the TTL, I'm not sure how the modem knows the difference and ignores the pfSense but not the laptop.
@Patch I do have all of my network behind the pfSense and the pfSense is directly connected to the modem. The image is just showing what the logical implementation of the modem is and what the IPs are. The modem is everything contained in the gray box including the built in 4-port switch. Hopefully that clears up what I was trying to show.
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
If it's not the TTL, I'm not sure how the modem knows the difference and ignores the pfSense but not the laptop.
A pcap started just before connecting pfSense to the modem might give us clues on that. I wonder whether the modem is juggling MAC addresses in some weird way, such that it presents x.x.x.186 on MAC m0:m1:m2:m3:m4:m5 to the laptop, but MAC m6:m7:m8:m9:m10:m11 to pfSense; pfSense probably responds to some ARP packets to which the laptop does not.
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
I do have all of my network behind the pfSense and the pfSense is directly connected to the modem. The image is just showing what the logical implementation of the modem is and what the IPs are. The modem is everything contained in the gray box including the built in 4-port switch.
-
I'm recommending you only connect pfsense to the internal switch in the Comcast modem/router (ie approximate as well as you can putting the Comcast modem/router in bridge mode).
-
Laptops currently with IP address 10.1.10.50 & 10.1.10.51 should be on the Lan not Wan side of pfsense.
-
Doing so simplifies network management.
-
-
@bPsdTZpW This is a great idea. I currently have a gigabit tap on order that I'd like to put inline to do some extended pcaps. (Why not use this as an excuse to buy a new tool...
) I'll post results when it comes in.@patch said in SG-2440 Upload Speed Limited After a Few Minuites:
I'm recommending you only connect pfsense to the internal switch in the Comcast modem/router (ie approximate as well as you can putting the Comcast modem/router in bridge mode).
This is what I'm already doing and have always been set up with.
Laptops currently with IP address 10.1.10.50 & 10.1.10.51 should be on the Lan not Wan side of pfsense.
The laptops in my diagram are only there to show what the IPs are when plugged into the modem. I do not have laptops normally plugged in. I would only plug them in for testing or to get to the modem web config (which for some reason doesn't work from the pfSense LAN... see above.)
-
@steve1515 said in SG-2440 Upload Speed Limited After a Few Minuites:
@bPsdTZpW This is a great idea. I currently have a gigabit tap on order that I'd like to put inline to do some extended pcaps. (Why not use this as an excuse to buy a new tool...
) I'll post results when it comes in.I'm looking forward to this data.
