Crazy low downloads speed, 980 Mbps upload, but it used to work!!
-
Greetings,
I'm new to pfSense. I went out and bought a 'AWOW' PC with 2 2.5 Gbps ethernet ports. Installed pfSense community on it, and got it up and going quickly. VERY Impressed, I was able to get 930 Mbps up and down, out of a max of 980 up/down that I was able to see with a direct connected PC to the Fiber. I have 5 STATIC Ips, and it was hard to find a firewall/router/gateway/anything that would support that. I had formerly (for 15 years!) been running a Netgear FVS338 VPN Firewall, and it failed once and I replaced it with the same unit and it was awesome. I upgraded my FIOS from 25 Mbps to 1Gbps recently, so it was time to replace the firewall.
Anyway, that was Thursday, and I only had one Static IP configured at that point. Sometime on Thursday (9/29) I got everything setup, it was routing my Nat'ed ports through the firewall and everything looked good.
I did have one time when my speed dropped to 25/25, and I was unable to figure out why, but rebooting the router fixed it... Weird, don't understand that.
I attempted to install darkstat on it, I read somewhere that it depended on another package 'trafshow' and I followed the install information for that, installing 'trafshow-5.2.3_31,1'... I was unable to get darkstat to work, port 666 just gave an error.
I gave up on that and went to bed, after confirming that all my port forwarding was working fine.the next morning, I ran a 'Speedtest.net' check, and got an unexpected result: Download was 25 Mpbs and upload was 930 Mbps on one machine, and on another machine (Mac Studio, M1 Ultra) it was 125-250Mbps / 930 Mbps. Looking at the status during that time, I noticed the max CPU it used was 8% or so, so whatever was causing the issue wasn't a CPU issue, and it couldn't be a ethernet issue because I was getting 930/930 the day before.
I've been fighting with this since then:
You can see that all the tests that I've run since 9/29 have been horrible compared to what I was getting then.
And of note, I tried putting a Mac Book Pro directly on the FIOS, setting the static IP and doing a SpeedTest.com test and boom: 980/980! So, the issue was not with the ISP.
Out of Total desperation, I booted off the install DVD (never got the USB sticks to work - in spite of trying 6 sticks with 7.9-64GB in size, and 2 different software packages to load it) and I reinstalled the ENTIRE OS on the disk - that will wipe out whatever is causing an issue! I then restored the configuration (I tried restoring the configuration from 9/29 as well - on the old setup) and that didn't work.
No luck. It was now up to 232/941 .. It got interesting from here... I tried running the test on two machines at once. One got 185/544 and the other 206/541, which implies that I was experiencing a total of 381 down, which would imply that there was some kind of limiting going on per machine?
I had no limiters configured in the system. The only thing I have is the Firewall/NAT configured with 4 aliased IPs, and about 6 Port Forwards from some external IP / Port 80 (or whatever) to some internal IP/80 (or whatever) to run a few websites.. No traffic on said websites yet.
I read about BufferBloat, and thought that maybe that could be it (although, it wouldn't explain how it used to work!), I tested for that on 3 machines, and I got a A a B and a C with the A being on the really fast computer I had (Mac Studio), the B being on a Mac Pro (Intel), and the C being on a 8700K Intel Windows Machine. I decided to follow the instructions here: https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
Oh, yeah, that fixed it good... Now I get 0 bytes/sec in both directions... I double checked every value it recommends.. something in that is causing it to pass NO data through the system! I've sense turned off the limiters - Btw: I did enter a value of 980 for the limit - as this is REQUIRED but NOT specified in the instructions... it says you need a value in the 'BW' (or something like that) field, and of course, you are sitting there wondering WTF is that?? I finally figured it out it was the Band Width, might have been nice to give that name instead of 'BW'...
Anyway, I'm back to being at a loss as to how this can be happening...
I'm also a bit at a loss if it is a problem or not, the https://www.waveform.com/tools/bufferbloat website is reporting 905 Mbps down on the machine that got the B, so is it really getting that speed? I mean, on the machine that has the C rating, speedtest.net is saying 250 (but was as low as 12 Mbps earlier in the day.. and 941 Mbps up)
Oh... One other thing that is really weird... on the 'C' machine I can no longer go to 192.168.1.1 - all the other machines can, but that machine is getting a timeout... Using the Brave browser.. It has a static IP configured via DHCP from the pfSense router...
Thanks for any help you can give me.. (I'm assuming the limiter(s) aren't working because I have multiple IPs, although all downloaded traffic should be coming from the public IP of the router)
-
@Traderhut It gives me an edit option, and I tried to fix the first line, but then when you save it... it says you can only edit for 3600 seconds (1 hour)... so, can't fix the posting - sorry..
@netgear - Stick to routers software, not doing so well on forum software. :-(
No clue how I got a scrollbar on the top, nor why the line comes up in a grey box.
-
If you have multiple IPs is there something that needs to be changed in the instructions to avoid the Bufferbloat? (to make it not block all your traffic?)
Maybe the rules need to apply to all external WAN addresses or something?
-
I'd be surprised if this is buffer bloat.
Check the output of
ifconfig -vvmat the command line.Check Status > Interfaces are there any errors or collisions shown?
Steve
-
@stephenw10 I think that I figured it out... I put the SWITCH back in front of the router - problem solved... Now getting 940/940.... OK, so the switch is 1Gbps, the router has 2.5 Gbps ports, the rest of my network is 1Gbps...
Here is my theory.... the FIOS will burst to > 1 Gbps, being as I know the other side is 2.5 Gbps (as a phone call can change my speed up to 2.5 Gbps if I wanted to), and of course, they are doing software throttling.. Which takes a moment to respond to the burst in traffic. So, what happens is it starts downloading the file. Sends a ton of packets and the router runs out of buffers... And it drops packets. Then the buffer drains out, but the other side is now waiting for a timeout on the dropped packet and depending on how things go, it only averages 25-250 Mbps/sec, spending 3/4 of its time waiting on timeouts for dropped packets.
But, by putting a 1 Gbps switch in front of it, it is effectively lowering the hardware speed that THEIR router can push data to me, and with it going never faster than 1 Gbps, my network/router never gets overloaded...
Still don't know what I did to make the change that was suggested in the link given cause all of the traffic to stop, but I assume that there was something..
So, I have a start on having my hardware able to handle the 2.5 Gbps in the future, with the router never going over 10% CPU, although, no way to know if the NICs can handle the throughput or not... although, not much point in putting in the 2.5 Gbps cards into the machine if it can't...
In any event, removing the switch in front was the only change I could think of that I had made between it working and it not...
-
Hmm, odd. I suspect something may have changed upstream then. Nothing you did should have affected it like that.
-
@stephenw10 It makes sense to me, their link to me can go up to 2.5 Gbps, and they use software throttling to slow down the link - which takes a bit to discover, so you will get a burst of 2.5 Gbps traffic, and that is over 250 M Bytes/sec, it won't take long to go from 0 to full on any buffers that pfSense has for inbound traffic, and the default only stores what? Maybe 1-10 Mb (1/25th of a second worth of data), Yes, some will be going out the other port (also a 2.5 Gbps port, but plugged into a 1 Gbps router, so it will only be able to push data out at 2/5ths as fast as it is coming in (40%).
And once those buffers are full, isn't it going to drop the next packet that comes in?
(To be fair, I'm not that familiar with the TCP protocol to know how it handles packet flows, From when I worked at Networth on the PowerPipes project, I recall the packets being 1450 or so bytes in size as a max, but I think that they got increased at some point over the years.. Even so, I assume there is a range of packets that are sent and you send ACK's (of some kind) for those that you got, or maybe only every so often, or maybe like ZModem, only a "Restart at packet X' when you realize you didn't get packet X. If the later, then you would get packet 1-X, X would be dropped due to overflowing the buffers, and you would timeout (1-5ms?) and request the Restart at packet X command and the data would start flowing again, but you would have a lot of packets > X still in the queue (with random drops of packets), and those packets would be sent, but ignored (or maybe saved to be used later, depending on the protocol.) and overall you would end up with a really slow connection...All evidence I've seen so far indicates that putting the switch in there is the solution, as then my ISP has a hardware limit on how much they can send to me, and their systems must be designed to handle that or they couldn't be hitting the max throughput very often.
-
Hmm, well if that is the case you might be able to set the NIC to just link at 1G only and be able to remove the switch.
