Traffic shaping best practice. Your default should be that all traffic is lowest priority, then you make rules that will lift specific traffic to higher priority. P2P uses ports all over the place. Just because the listening port is specified doesn't mean it isn't free to use others. I've tried.

I'm seeing the traffic graph issue as well using a PRIQ shaper. Although it became most pronounced (e.g. off by more than 50% - shows ~4 Mbps for an 11+ Mbps flow) only after I disabled all of the scheduler options (e.g. Codel, RED, ECN, etc). I mentioned this issue at https://forum.pfsense.org/index.php?topic=115862.0 but figured I should post it here as well in case the extra data is usefull. WHen Codel or ECN were enabled, the traffic graph was much closer (within 1 Mbps or so) to being correct (if not exactly correct).

Is there an open bug report for this in redmine? If not, should I open one?

So with a bit more testing, turning off CODEL (along with all the other scheduler options) gets me the best bandwidth: between 11 and 12 Mbps.

Doing so, however, seems to badly break the traffic graphs. They now display radically lower throughput than I'm actually getting. See attached graph showing ~4 Mbps when I'm actually getting a fairly steady 11+ Mbps up. Maybe related to https://forum.pfsense.org/index.php?topic=89247.0? Interesting that the issue is most pronounced when no special scheduler options are enabled.

WAN_Traffic.png_thumb
WAN_Traffic.png

Oops, miss typed.  I have max at 400 - so 400MB.

It looks like it goes into memory, but not on the SSD.

After the last post I noticed my WAN was 100Mb. I set it to 99Mb and it worked.. I was like… Hmmm.. I'm using percentages and just decreased the rate and it worked. So I tried 98Mb and I got the error again. If you look at the actual numbers for the parent queues, they only add up to 99Mb even though the Interface is 100Mb. That's because the UI does not like real numbers. So setting it to 99Mb was fine. But trying to set to 98Mb was right-out. Tallied values are now 98.01Mb. If only I could use real-numbers  :p

queue root_igb0 on igb0 bandwidth 99Mb priority 0 {qACK, qUnclassified, qClassified}
queue  qACK on igb0 bandwidth 19.80Mb qlimit 1024
queue  qUnclassified on igb0 bandwidth 29.70Mb {qUDP, qDefault}
queue  qUDP on igb0 bandwidth 13.07Mb qlimit 1024 hfsc( codel linkshare(16.34Mb 5 13.07Mb) )
queue  qDefault on igb0 bandwidth 13.07Mb qlimit 1024 hfsc( codel default )
queue  qClassified on igb0 bandwidth 48.51Mb {qNormal, qHigh}
queue  qNormal on igb0 bandwidth 21.34Mb qlimit 1024 hfsc( codel )
queue  qHigh on igb0 bandwidth 21.34Mb qlimit 1024 hfsc( codel linkshare(26.68Mb 5 21.34Mb) )
queue root_igb1 on igb1 bandwidth 99Mb priority 0 {qACK, qUnclassified, qClassified}
queue  qACK on igb1 bandwidth 19.80Mb qlimit 1024
queue  qUnclassified on igb1 bandwidth 29.70Mb {qUDP, qDefault}
queue  qUDP on igb1 bandwidth 13.07Mb qlimit 1024 hfsc( codel linkshare(16.34Mb 5 13.07Mb) )
queue  qDefault on igb1 bandwidth 13.07Mb qlimit 1024 hfsc( codel default upperlimit 23.76Mb )
queue  qClassified on igb1 bandwidth 48.51Mb {qNormal, qHigh}
queue  qNormal on igb1 bandwidth 21.34Mb qlimit 1024 hfsc( codel )
queue  qHigh on igb1 bandwidth 21.34Mb qlimit 1024 hfsc( codel linkshare(26.68Mb 5 21.34Mb) )

Bocking "all" is going to be pretty difficult but you can:

control mime type apply such kind of approach.

@Harvy66:

I don't think shaping works on a LAGG.

That's correct. A LAGG interface on its own does not support ALTQ shaping. You can use a VLAN interface on top of LAGG if you require shaping. You have to setup the VLAN tagging in the switch but it should only be a few extra clicks or config entries on most switches.

@jetblackwolf:

Thanks for the feedback. Trying not to hijack the thread, just meant to pop in and try to offer some information.

I'll gladly try and help with a guide. (at least for HFSC) Trying to get the correct setup going first before I created a big thread that was full of misinformation. Not on purpose mind you, along these months I have closed the book on this many times believing I understood what was going on and then started all over. So I am in an odd position where I can spot a bad setup now based on all of my tests and what I have concluded on….but not confident enough to offer up any kind of guide on my own.

I think some questions I would love to see answered for HFSC/Codel are what happens when new streams come into a queue? Do they abide by the M1/D while other streams have already met the M2? And where would the bandwidth come from? M2? Or dig into another child queue? Or does the queue literally fire off one M1/D check on first use of that queue and then potentially sits there in M2 mode until the queue returns to an idle state, to then repeat the process again? Many areas of the papers I read went right over my head, not a PHD by any stretch.

I spent time trying to assign the priority for the HFSC before noticing it doesn't actually seem to be a part of the queue documents, at all. This is confusing because there is a note in the GUI that says it sets priority on packets during overload (for HFSC). Yet HFSC only has bandwidth and time variables. Spent a bit of time on this before finding out it does nothing......at all. So even if the GUI was cleaned up and only the proper options provided for the selected queue, it would probably be less confusing. ::)

Why I believe the drops are still occurring on my end is related to how multiple LAN queues are being hammered at the same time, even though all upperlimits are correctly divided. I will look into that burst comment Harvy66. (and yes I am using CODEL per your findings in older forum posts)

Regarding HFSC, please post your questions in my HFSC explained - decoupled bandwidth and delay - Q&A - Ask anything thread. That thread also has links to the best HFSC documentation that I came across while researching HFSC.

You are not alone in your confusion… :)

Traffic shaping affects the entire interface. A VLAN is an interface. If you set a VLAN to be 15Mb/s, then ALL traffic will affected. If you want to shape it to recognize LAN-to-LAN flows, then you'll need to configure the queues and firewall rules that way.

quick checked or no checked they still evaluate to last one wins but other than that any traffic from lan client to pfsense gui just doesnt goto qlink in my case

Both the bandwidth and borrow elements were missing from CBQ queues. Those have now been restored and should appear in the next snapshot.

https://forum.pfsense.org/index.php?topic=79589.0

Same difference. In a simplified sense, queues don't care what traffic goes through them. Of course there are exceptions, but they're agnostic about the traffic.

@deagle:

I would like to classify streaming, then make sure each tenant gets a fair amount bandwidth while giving regular browsing priority. I'm open to suggestions if you have some.

Interesting thing to know about networks, assuming not super slow like below 10Mb. At any given time, there are only dozen or so flows of packets in the buffer, and nearly all of the packets are from the single digit heaviest of data flows. This rule of thumb applies from a 133Mb link serving 500k active flows, to 10Gb links also handling 500k flows.

This means a few things are happening

When a tail-drop buffer is full, the smaller flows get hurt the most because they lose the most packets percentage wise. A select few flows monopolize the network and it's hard for any other TCP flows to get in edge wise

CoDel and FairQ break this up. In the case of Codel, it's primarily head-drop and is a time based buffer. This means a few things

There's almost always room in the buffer, allowing small flows to make it in instead of getting tail-dropped CoDel is most likely to drop a large packet from a fat flow

Unless you have a really low bandwidth link that is completely overwhelmed, CoDel will help maximize bandwidth, minimize latency, minimize loss, and redistribute bandwidth by dropping packets from the fatest of flows freeing up bandwidth for other flows to move in.

I recommend trying this approach first. fq_Codel and eventually Cake will be much better at this. fq_Codel already works wonders, but Cake has a lot more features, but it's also taking longer than expected because of performance regression caused by so many features added. Unfortunately PFSense only has Codel right now, but that alone is the 80/20 rule.

@Berend:

It also looks like the "Kill States" button isn't work either anymore.

The in direction and translated destinations wasn't working.
https://redmine.pfsense.org/issues/6530
https://redmine.pfsense.org/issues/6531

You can easily accomplish your goal with HFSC traffic-shaping queues. 1 queue per tenant, then set the link-share & upper-limit values to the appropriate values.

If you need to group certain tenants you can create a parent queue with the appropriate tenant queues as child queues.

Forget about limiters.

If it's a specific IP address, you could just use priority queue and create a firewall rule that assigns all traffic going to that destination to be placed in your high priority queue. Priority queues can cause issues in some cases. I prefer to use HFSC. Then you just assign a minimum amount of bandwidth, and it will guarantee that all of that traffic will get at least that amount of bandwidth.

Thanks Nullity and Harvy66 for reply.

You are right, I want fair usage of bandwidth. As of now I have setup equal share and it is working.

What about Penalty Box? Will that work?

Can you please guide me for setting up users' own queue? Just some link will work. Thanks.