Rate limit not honored? - Solved, blame BitTorrent and high latency peers
-
Strange issue. As far as I can tell with doing speed tests and other basic tests, the rate limit on my LAN interface is honored, yet when I queued up a bunch of Linux ISOs in BitTorrent, I was getting above the rate specified. As far as I can tell, only BitTorrent causes this issue, but there were a lot of connections to many peers.
I have 100Mb/100Mb and set my LAN rate to 95Mb to smooth out my downloads by delaying or dropping packets from high bandwidth downloads. This keeps others from sending data too fast.
Here is a traffic image showing pretty much 100Mb coming in. You can actually see when I changed my rate all the way down to 80Mb, only then did it finally get my bandwidth lowered.
Here is a graph of my max LAN out being well above 95Mb. I even set it to 90Mb for a bit, and yet it kept coming in.
Of course this made my quality graph mad when my traffic graph was showing me 99.5Mb/s coming in on my 100Mb connection
Like that 0.14ms min ping to my ISP? No buffer bloat here. 8)
Mmmmm 98.5Mb/s 10 minute sustained average at 8:50pm-9pm? Only $90/month :-)
-
Do you have a port forward for your BT port? If so, you have to put that through the limiter too.
-
I have the entire LAN interface set to 95Mb. To answer you question, yes I have a port forwarded, but I'm not aware how your rate limit a port. I only see ways to rate limit interfaces and queues. I wonder if the TBR needs to be tweaked. I can't find any information about it.
-
You have to place the traffic in the queues using firewall rules. Simply defining the queues does nothing. Many times people place the traffic from LAN clients into the queues on the LAN firewall rules or floating rules on WAN out. But with a port forward the states are created using that rule instead so the traffic isn't placed into the proper queue. This is almost always missed when BT is the target.
When testing queues, I like to set a low limit, sat 10Mbit/s on a 100Mbit/s circuit. Then I KNOW I'm seeing the shaping not something else. When you get the hang of it, you tweak it to maximize what's available.
-
Perhaps try to also limit your outgoing ACK throughput related to torrents?
-
You have to place the traffic in the queues using firewall rules. Simply defining the queues does nothing. Many times people place the traffic from LAN clients into the queues on the LAN firewall rules or floating rules on WAN out. But with a port forward the states are created using that rule instead so the traffic isn't placed into the proper queue. This is almost always missed when BT is the target.
When testing queues, I like to set a low limit, sat 10Mbit/s on a 100Mbit/s circuit. Then I KNOW I'm seeing the shaping not something else. When you get the hang of it, you tweak it to maximize what's available.
Here are my rules pertaining to BitTorrent. As far as I can tell, when looking at PFTop, P2P traffic is getting thrown in the P2P queue and if it isn't for some reason, it is definitely going into the Low Queue because of my catch all floating rules. These floating rules are both LAN and WAN interfaces for Any direction.
BitTorrent is configured such that on out going connections, the source port is also the same instead of random. I'm not sure how that interacts with NAT since NAT typically randomizes ports, but since the floating rules also look at the LAN interface, which is before the NAT, it should correctly see the source port and match the state. No matter what, all traffic falls into some queue because of my catch-alls.
-
Perhaps try to also limit your outgoing ACK throughput related to torrents?
I guess my main point is I have the LAN interface set to 95Mb, my expectations is that the LAN interface will NEVER, under any circumstances, go above 95Mb/s egress. Isn't that the whole point of limiting the interface itself and not the queues directly? Anyway, child queues may not be greater than the root queue, which makes no sense.
I had a thought, I think I have "Unknown UDP" set to have 5% real time. If real time is coming from 1Gb and not 95Mb, this would cause an issue. Torrent is a bit special in that it has a lot of on the side UDP protocols and can use random ports that I can't control. The bulk of Torrent traffic is on known ports, but some is on these random ports.
I'll give that a try when I get home, to disable realtime on unknown UDP.
-
I played around with a few rate limits, here are the results when using BitTorrent.
Started with the Rate limit set to 50Mb, the actual speed allowed through was almost 70Mb/s.
I then changed it to 25Mb, the actual speed allowed was 35Mb
I then set it to 80Mb, and you can see it jumping up to 101Mb, at which point I start seeing periodic packet loss because I only have a 100Mb package.
What next should I try or what information shall I divulge?
-
Looks like it's working pretty well. Note that it's not magic. All it can control is how fast information is transmitted out an interface.
-
And it's transmitting 101Mb/s out of my LAN interface, that's my issue. 0 packets are being dropped, all packets coming in the WAN go out the LAN, and it's well above my set rate limit.
-
Dude. BitTorrent does EVERYTHING it can to keep from being throttled.
I'd set my queue to 75Mb and move on. Open a bug report if that's not enough.
-
Dude. BitTorrent does EVERYTHING it can to keep from being throttled.
I'd set my queue to 75Mb and move on. Open a bug report if that's not enough.
98 times out of a 100, the fault will be me, one of those will be something strange, and the remaining 1 might possibly be PFSense.
This was driving me craaaaazy!
I got this now. When I glanced at the LAN egress, it looked to be about the same as the WAN ingress, but that's only because I glanced. I decided to study it for a while, along with the queue status. I watched all 3 (WAN, LAN, and Queue) for about 10 minutes…. I was determined that this was my fault or something strange at play.
Here's why I was convinced it was me misconfiguring something in PFSense.
- About 98Mb/s coming in
- "close to" the same going out the LAN
- No drops in my queues
- No packets being blocked
- When I do speed tests, everything works perfectly
Well, if 98 is coming in and 0 packets are being dropped and 0 being blocked, then 98 must be going out... nope. About 90. WTF?! How can that happen?
What kind of packets can be effective blocked without being logged as blocked? Retransmissions. Holy crap batman, did I hit the mother load. I figured that if I was rate limiting PFSense to 80 and seeing about 90, then I had about 10% extra traffic on average. I could choose a TCP stream at random and almost always find one with huge amounts of the captured packets being me receiving dups and replying.
I stopped BitTorrent and decided to trace route some of these offending IP address. A lot of them where Cox cable customers and they had massive packet loss, latency, and jitter in their own networks. I was seeing trace routes jumping between 70ms and 150ms, to Cox hops, and much worse pings to the actual destination IP. Mind you, my ping to Cox edge routers were fine, it was the routers inside of Cox network that kept getting worse the more you went down.
I'm sorry if I have wasted some people's time, but I am a bit obsessed with knowing "why". This was a crazy case, but I feel confident that I finally figured it out.
Edit: In case not immediately obvious. Retransmissions caused by high latency create dup packets. In this case, a lot of dup packets.
-
No waste. Interesting find.
