Bad bufferbloat scores with "bandwidth limit per IP"
-
Just wanted to shared my experience with "bandwidth limit per IP" limiter(aka no child queue limiter). I spend a couple of days testing multiple options.
Setup is like so:
200Mpbs < WAN firewall LAN > 1000Mbps < port switch > 100Mbps < client host >
- per host limiter with a peak bandwidth of 80-95Mbps
The reasoning here is that I want to shape per host(IP) bandwidth to a level which is less than the 100/100Mbps port they are connected to. If I fail to do this this manifests on the access port as "output drops", which is normal, because that would be the bottleneck in this setup and the point where the switch starts to indiscriminately drop packets.
Option A. So my idea was to throw in a per IP/Host shaper on the LAN side of the firewall and cap each IP/Host to 90-95Mbps. Doing so, of course results in no "output drops" on the switch port. So far this is great and on paper it looks great. However when You run some flent or bufferloat tests, You'll quickly notice that with the limiter on the LAN, You'll struggle to get anything better than "C". I tried multiple schedulers and AQM combinations.
Option B was to just stick the shared limiter for all the hosts to 200/200 and use a queue. This in combination with fq_codel results in tripple "A+" most of the time. Running no limiters at all results in "A/B" most of the time. With "output drops" on the switchport of course.
In both cases the limiter is applied via a floating rule with an alias(which covers the network in the LAN). I can post a configuration, but it's trivial to replicate.
-
i've only experienced 'output drops' on crappy old 10/100 gear ....
i've never seen it happen on useable gear from this decade -
Well You are not wrong.
It's a Cisco 3750 stack and as the 2960 and 3560 before it, they suffer from extremely small port buffers, which Cisco claimed are not a problem, but silently fixed in future switch families.The main question still remains as to why limiter with no queue with no drops on the switch port performs worse than limiter with queue and drops on the switch port.
-
@xciter327 said in Bad bufferbloat scores with "bandwidth limit per IP":
they suffer from extremely small port buffers
Simple solution - would be to replace them ;) Wouldn't it... I mean really who connects at 100 these days anyway..
-
You would be surprised. Regarding replacement, I've got hundreds of these(because "we only work with Cisco" and "we don't buy new switches, because there is no point"), so replacing them would not really be an option. For the most part it works great and I do get consistent triple A or even A+ most of the time with the shared limiter with queue. The output drops show up as discards on our monitoring and that triggers my OCD. :)
-
@xciter327 said in Bad bufferbloat scores with "bandwidth limit per IP":
"we don't buy new switches, because there is no point")
There is no point if your cheap I guess ;) And you always want to be behind in tech..
But ok.. So your saying your an enterprise and you have 100's of these cisco - and you only buy cisco... But they have to be OLD so your getting them for cheap..
So your local speeds your clients are limited to 100 -- the local network must scream... So with 100's of switches I have to assume 100's of clients if not more.. And you have 200mbps wan and your worried about some buffer bloat.. What is that going to matter when you have 100 users trying to use the net at the same time sharing 200? So they are not going to get anywhere close to your limited 90mbps speeds.. Unless there were like 2 of them..
-
Fully agree with you. To clarify there are not hundreds of switches on this 200Mbps link, just overall in the network.
In this particular example the 200Mbps link will be shared by 50 100Mbps clients. So essentially what I want to do is to have a limiter per IP/Port to shape them to 85-90Mbps(which does solve the output errors) and have another limiter which splits the 200Mbps of bandwidth evenly between the 50 clients(if they decide to request at the same time, which never happens).
-
have you considered thinking the other way around?
instead of limiting the maximum, there might be a way to provide a minimum bandwidth to each client. (i'm no shaping expert, so perhaps that is impossible)
-
So these 50 clients do not talk to anything else on the network... If your saying the buffers on these ports of these old switches are horrible... limiting them at your edge is not going to have effect on local traffic causing these output drops on the ports.
Why would you not just rate limit them down to 80mbps at the switch port to fix your OCD problem of drops on the ports..
And use your overall queue to get you the wan shaping you want.
-
There is no local traffic. The switch uses pVLANs to isolate them. The only way they can communicate locally is via the gateway(aka pfsense). I'll give the rate limiting on a port a go and see how it does.
My idea was that the the firewall being the the "edge" of this network is in a better position to do the shaping with one of the new fancy AQM/schedulers rather than the Cisco switch access port which drops indiscriminately to achieve a certain level.
