I agree that it is ugly and is not by any means my preferred method but It appears that it might be my only choice in this particular scenario. Thank you for your insight.

@uptownvagrant Thanks. If I’m understanding the post you linked I should have the same settings as described there now that I have changed to tail drop on in/out pipes and associated queues.

This absolutely works!!!!!!!

Thanks to you both so very much.

I agree. Not an issue of the algorithm but the implementation. There seems to be no documentation on how to properly configure ipfw to make proper use. ipfw is the issue. It has a bunch of knobs and those can have negative interactions that ruin the benefits of fq_codel.

@jimp I'm also seeing ICMP echo replies dropping when NAT is enabled, a limiter is being used, and the limiter is loaded with traffic. This is also documented in Playing with fq_codel in 2.4 and bug 9024.

Issue 1:
Using limiters on an interface, with outgoing NAT enabled, causes all ICMP echo reply traffic to drop when the limiter is loaded with large flows. I can reproduce this issue with the following configuration.

limiters created (any scheduler). One limiter for out and one limiter for in. create a single child queue for the out limiter and one for the in limiter. floating match IPv4 any rule on WAN Out using the out limiter child queue for in and in limiter child queue for out. floating match IPV4 any rule on WAN In using the in limiter child queue for in and out limiter child queue for out. load the limiter with traffic (most recently I've been using a netserver v2.6.0 on the WAN side and a Flent client on the LAN side running RRUL test) start a constant ping from the client to the server during the RRUL test

Both the flent.gz output and the constant ping will show a high rate of ICMP echo reply packets getting dropped. If NAT is disable you will not see ICMP echo reply drops. If NAT is enabled but the limiter is not being loaded with traffic you will not see ICMP echo reply drops.

Yeah, see, it balanced (kinda) all internal interfaces to be at 20%. I don't know how exactly it did that, one of the pfsense guys most likely does.

I would build the limiters by hand - you've only got 4 internal interfaces. You can set hard limits you want on each interface (I would skip the WAN and LAN2 interfaces) to max out at, say 80% or 85% of your WAN pipe bandwidth. You might want to allow wifi access points on LAN3 be a lot slower than that.

I guess the answer to just how slow to make them depends on what you're doing on those 4 interfaces - LAN1 thru LAN4.

You know where to make these, right? Under Firewall -> Traffic Shaper -> Limiters. At the least, you could make 2 limiters, 1 for upload, 1 for download, then test them on the "allow any to any" firewall rule on an interface. That would let you see how it works. For more fine grained control for the other interfaces, simply make more traffic shaper limiters.

As an example, I've got a Guest VLAN for wireless access points. I have set 2 limiters - 1 for download, 1 for upload. In there I allow 10% of my entire WAN pipe and it works really well.

Hope that helps!

Jeff

I solved it by checking the Per-user bandwidth restriction and leaving the fields (Default download/upload Kbit/s) blank.

There is no local traffic. The switch uses pVLANs to isolate them. The only way they can communicate locally is via the gateway(aka pfsense). I'll give the rate limiting on a port a go and see how it does.

My idea was that the the firewall being the the "edge" of this network is in a better position to do the shaping with one of the new fancy AQM/schedulers rather than the Cisco switch access port which drops indiscriminately to achieve a certain level.

Thanks about the background info for the shaper function.

If I would add VIPs from the 2nd interface to the 1st, I have first of all disable the 2nd interface so that I will have not after commit on both interfaces the same IP's, then I have to check my NAT and LAN rules, having the correct destination addresses / gateway address in use. OK, should be possible to do.

What do you mean with the 2nd hint:
"Or better yet, just use one /29 on WAN and have them route the second /29 to the firewall address in the first."?
Can you explain a little bit detailed please?

Thanks for your help.
Frank

No problem. My line is 220/6 Mbit. I don't want to limit clients by bandwidth, I wan't the line to be utilized fully. I only wan't to prioritize traffic by categories and eliminate bufferbloat.

I'm currently experimenting with tagging all VLANS at a single physical interface and I try to limit that single interface in traffic shapping but it's not working, I'm still hitting the 200 Mbit limit with VLAN to VLAN traffic.

This is my current VLAN setup: https://i.imgur.com/HreJkxG.png
This Shaper setup: https://i.imgur.com/9O51sjd.png

https://www.netgate.com/docs/pfsense/book/trafficshaper/index.html have fun reading it.

It is for matching. pfSense cannot set DSCP.

@derelict i get it. It's just something that is used often in our world

IIRC, shaping over multiple LAN interfaces is either not fun to configure or doesn't work at all. I don't remember which.

@mattund said in Limiters and floating rules:

@revengineer

I count it as "in the noise" per-say, and let it pass my mind. I find ICMP traverses the network leisurely anyway, and besides, I haven't found a way to NOT drop the traffic -- if ICMP is getting dropped a lot of other problems can seep in...

Great, thanks. I made the change and confirm that I can ping my cable modem connected to WAN even under full bandwidth load.

I have had a similar thing happen to me. I had an existing setup with limiters already applied to rules and trying to change from either QFQ or FQ_CODEL scheduler to PRIO caused a kernel panic. I was only able to isolate it to something in traffic coming in off the LAN as disconnecting the LAN stopped the system from panicing on reboot an allowed me to restore the config.

No idea what the traffic might have been causing the issue. Suffice to say I will be keeping clear of the PRIO scheduler.

Tried applying to LAN with same result
Tried lowered quantum to 300 (not sure which value i should use but found another topic with one using 300)

I have to read more about the subject before I can answer the rest. I just tried it out for fun to see what it could do. My connection does't suffer that much from bufferbloat but I have a few friends that does and wanted to see if this could help them out at some point.