1Gbit Symmetrical Upload Slows to 80Mbps
-
I'm kind of at a loss on how to further troubleshoot this and hoping somebody has some suggestions to try. I can run speed tests a couple-three times and max out the link (900u/d) and then randomly can run it again from any different and speed drops down to around 600 or so and 80upload with packet loss on the gateway.
Basically, I'm a fairly new fiber internet user (woot) that really wants to run pfsense at home, but having some abnormalities in regards to speed tests. Here is a list of everything I've tried at this point and simple layout information:
ONT > pfsense (physical not virtual) > switch > end user device/server/etc.
I don't think it's a lack of physical hardware resources available as I've tried two completely separate physical devices running pfsense and OPNsense and the results are the same. One was a Dell 5050 i5-7500cpu/8gb ram with a pulled server Intel igb quad port nic. The other device is a older Dell R610 pizza box with broadcom 4x nics and dual Intel(R) Xeon(R) CPU E5620s w/ lots of memory.
The config is pretty vanilla out of the box (no packages since I thought one was causing the issue) outside of configuring interfaces, dhcp server, dns, and firewall rules to allow traffic on each interface via a copy of "Default allow LAN to any rule". I tried settings here for each box and their respective NIC tweaking https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#intel-igb-4-and-em-4-cards
I've also tried with and without disabling Hardware Checksum Offloading on both platforms. CPU never gets very high on either box when running more speed tests.I've tried different/new patch cables and different switch (one was layer 3 and the other is layer 2) and didn't seem to make a difference. We're also only talking a small amount of devices (10-16) even going through the firewall, so the states are super low in use.
I contacted tech support for ISP and asked is it perfectly fine to run straight from the ONT to my own firewall and he said yes and their little gateway/router device isn't doing anything special. I plug into their device behind switch or not, and taking firewall out of the equation and speeds are fine 24x7. I think I even tried just feeding my WAN port on pfsense from their box (which isn't in bridge mode, so it's giving out a 192.168.1.x based address) and speeds seemed to be okay.
I'm not doing any crazy vlans or anything on pfsense, although each interface is using a different network address.
-
I'm assuming that you are running a speed test from a computer, like at dslreports.com?
You may be having the same issue I did, originally with ATT UVerse and now with Spectrum 400meg down cable (no fiber here {SIGH}). That is, for light work, things ran OK, but the moment anything taxed the internet connection like a speedtest, I could see high latency increase while watching the gateway monitor from the STATUS PAGE. It would progress as the test ran, from latency to packet loss to the network going 'down'. The test would complete and after a minute or so, the gateway monitor would return to normal. I cured that by using a limiter (FIREWALL/TRAFFIC SHAPER/LIMITERS). There was some trial and error, and searching here for info on Spectrum and traffic shaping along with watching a few Youtube videos (Lawrence Systems has a good one IMO), but once I dialed it in the issue is solved for me. The speed tests that used to crash my connection no longer present any issue so I know that with normal use I should not have any problems (and I haven't).And yea I had gone through the whole swap NICs, switches, going direct to the router, swapping cables, two different PFSense boxes before admitting it's not hardware. It's having to roll up my sleeves and setup the limiters. Turned out to not be so difficult, mainly running a speed test over and over and tweaking only one or two settings on the limiter till I got the best speeds without seeing the latency climb to a point where I got packet loss.
-
@tzvia I think I had tried limiters based on Lawrence's videos, but removed them after it didn't make any difference (too many reloads ago to remember at this point). I can login to the ISP gateway/router and there isn't any type of limiter or QoS in place there, so not sure why their little probably cheap device can run speeds fine with no hiccup vs pfsense having the quirk. I'll go through all of it again and see if it makes a difference and report back.
-
@gdadkins Is it possible that the ISP router is pre-setup for their service, turnkey?
And yea with all the choices in the limiters it can seem daunting to mess with, just getting in a ballpark where your settings make an effect. I wouldn't bother with the wizards, just make two queues with codel and fq-codel (unless others here with the same ISP/service recommend something else). I had to set a QUEUE LENGTH that was recommended here then tweak it a bit. Your issue sure sounds like something a correctly tuned limiter would fix.
Sounds like you have done all the hardware swapping I had gone through. When I switched to much faster cable service, I removed the limiters that were setup for the old service, and my problems returned. Thankfully there were suggestions here that got me 'close enough to spit' and I was able to tune the problem away. -
The ISPs router almost certainly does not monitor the upstream gateway so it will just keep sending whatever the latency is. When you put pfSense behind it by default pfSense monitors it's own WAN gateway which will then be the ISPs router. You are not likely to see any significant latency locally there which might explain why you see no problems.
You should certainly try disabling gateway monitoring action or even monitoring entirely. I would not expect that to make any significant difference with only one gateway though.
However if it does you can tune the gateway monitoring for your connection far better than the default values. At least set it to monitor something further upstream.Steve
-
@stephenw10 Although not ideal, after getting login credentials to the ISP provided router, I moved everything behind their device (which is a calix gigacenter 844e-1 which is actually not a bad device) and speeds are running normal with no weird latency when upload was seemingly capped. Their device provides the option to place my pfsense box in a DMZ, so this allows me to open ports (ie 443) and route things like I need to.
I really wanted to figure out why it was acting the way it was directly connecting the the ONT box, but I'll roll with this for now as it seems there is something upstream that is hampering devices that aren't isp devices. Thanks for the information from both of you guys.
