HFSC/CoDel for 40 devices

moikerz

I have an office with about 40 devices, with a 6/6 access (yay T1..). All I want to do is de-prioritize SMTP, and prioritize access to one website, while not letting any one user monopolize the connection. Directing traffic to the queues is not an issue.

I've had PRIQ working somewhat well, but lower queues were sometimes starved for bandwidth. (also, dslreport's bufferbloat was always an 'F').

I figured I could try pure CODELQ, but then I was reading that it doesn't perform well for multiple simultaneous users/threads. It gave me an instant 'A' for bufferbloat, but if it would be problematic for 40 devices, I'd prefer not. Is this true?

If so, then I guess I'm left with HFSC. But I'm getting confused. I can understand the parent/child relationship, but I can't for the life of me figure out Linkshare's relationship to Upperlimit/Realtime … ???

Example time:
Let's say I have a parent, with two children (HighPriority, LowPriority). I don't care about burst (m1,d); only about m2.
I want HighPriority to have minimum 30% bandwidth, maximum 80% bandwidth.
I want LowPriority to have minimum 10% bandwidth, maximum 40% bandwidth.
So HighPriority's Upperlimit = 80%, Realtime = 30%, Linkshare = ??
And LowPriority's Upperlimit = 40%, Realtime = 10%, Linkshare = ??

WTF goes in Linkshare for the two children? ??? Am I not thinking of 'maximum' correctly?

Harvy66

The text is missleading or possibly wrong, but I have seen similar descriptions from tutorials. The issue is dumbing down the descriptions loses information that makes things more confusing when trying to reason through the more powerful features like Upper Limit and Real Time

Upperlimit and linkshare are both relative to their parent queues, Real Time is relative to the root queue. As a general rule of thumb, I do not use realtime. It creates a mess of things by overly complicating simple issues.

I would just stick with Upper Limit and Bandwidth and use percentages. Set your Bandwidth to the minimum percentage you want to have and remember it's relative to the current parent queue. eg If your parent queue has a bandwidth of 50% and your child queue has a bandwidth of 50%, then your child queue has an effective bandwidth of 25% relative to the root queue.

Upperlimit works the same way. If your parent queue can only have 80% of the root queue and your child can only have 80% of the parent queue, then your child queue can only have 64% of the root queue.

moikerz

That's good information, thanks Harvy66! Realtime makes perfect sense; it would be nice if the UI could just say what each block is relative to (parent/root).

So with the concept of "borrowing":

1. using Bandwidth=minimum, and Upperlimit=maximum, does that imply that the child can borrow from the parent until it reaches the child's Upperlimit? Nothing else to set?

2. using percentages, does the Upperlimit of Child1 + Child2 need to equal 100%, or can it be more?

Harvy66

@moikerz:

That's good information, thanks Harvy66! Realtime makes perfect sense; it would be nice if the UI could just say what each block is relative to (parent/root).

So with the concept of "borrowing":

1. using Bandwidth=minimum, and Upperlimit=maximum, does that imply that the child can borrow from the parent until it reaches the child's Upperlimit? Nothing else to set?

2. using percentages, does the Upperlimit of Child1 + Child2 need to equal 100%, or can it be more?

Correct. The percentage can be misleading, but it really is just a short-hand to place relative fixed bandwidth amounts, so it's pre-computed, not dynamic. As long as the child is not at it's upper-limit, it follows normal distribution rules.
Correct. Upper limits restrict, so they can be more than 100% among queues, but no more than 100% for a single queue

Interesting to note is that HFSC effectively distributes bandwidth in ratios. If I set one queue to 1% and another queue to 1%, and they both try to use all of the bandwidth, they will get a 50/50 split.

Nullity

Forget about real-time. Like Harvy66 said, ocus on link-share (and maybe upper-limit on download traffic). Link-share is just a ratio, not an absolute, so pay attention to the proportional relationships of the queues. Keep your rules and queues simple.

Sadly, upload & download need to be treated differently. I would not use upper-limit on upload, because the router can throttle the LAN clients almost immediately to make bandwidth available for VIP traffic.

Download needs to be preemptively throttled, sometimes as low as 60% to account for the exponentially larger delays between when you request a slower download and when the download actually slows down.

moikerz

So here's what I've currently got:

WAN, HFSC, Bandwidth: 5Mbps
- qInternet, CoDel, Bandwidth: 5Mbps
- qDefault, Default, CoDel, Bandwidth: 10%
- qOthersHigh, CoDel, Bandwidth: 20%
- qOthersLow, CoDel, Bandwidth: 5%

LAN1, HFSC, Bandwidth: 900Mbps
- qLink, Default, CoDel, Bandwidth: 895Mbps
- qInternet, CoDel, Bandwidth: 4Mbps, Upperlimit: 4Mbps
- qOthersHigh, CoDel, Bandwidth: 10%
- qOthersLow, CoDel, Bandwidth: 5%

LAN2, HFSC, Bandwidth: 900Mbps
- qLink, Default, Bandwidth: 895Mbps
- qInternet, Bandwidth: 1Mbps, Upperlimit: 1Mbps
- qOthersHigh, CoDel, Bandwidth: 10%
- qOthersLow, CoDel, Bandwidth: 5%

So, here's my questions and notes:
1. So I should remove the WAN-qInternet Upperlimit value? Not sure I understand your reasoning, Nullity.

2. For LAN1-qLink and LAN2-qLink, is this bandwidth correct? Or should it be my Upload bandwidth?

3. LAN1-qInternet + LAN2-qInternet = 5Mbps, is this the appropriate way to shape the LAN2 speed?

Harvy66

You don't need an upper limit on your WAN because the interface is already limited to 5Mb total, Your LAN interfaces are not.

2+3) Because PFSense does not allow sharing bandwidth among interfaces, you are correct that you need to split the bandwidth between LAN1 and LAN2 in fixed amounts. PFSense shapes bandwidth going out because technically you can't shape bandwidth coming in. Your LAN interfaces represent your download.

moikerz

I understand about have LAN1 and LAN2 Bandwidths set as my desired download speed.

But what does that mean for LAN1-qLink and LAN2-qLink? Since they are my default queues for LAN, it sounds like I should remove the qLink parents completely, and make new qDefaults as children of qInternet, correct? (With the obvious side-effect of limiting inter-vlan comms to whatever download speed I configure for the LANx parent)

WAN, HFSC, Bandwidth: 5Mbps
- qInternet, CoDel, Bandwidth: 5Mbps
- qDefault, Default, CoDel, Bandwidth: 10%
- qOthersHigh, CoDel, Bandwidth: 20%
- qOthersLow, CoDel, Bandwidth: 5%

LAN1, HFSC, Bandwidth: 4Mbps
- qLink, Default, CoDel, Bandwidth: ???Mbps
- qInternet, CoDel, Bandwidth: 4Mbps, Upperlimit: 4Mbps
- qOthersHigh, CoDel, Bandwidth: 10%
- qOthersLow, CoDel, Bandwidth: 5%

LAN2, HFSC, Bandwidth: 1Mbps
- qLink, Default, CoDel, Bandwidth: ???Mbps
- qInternet, Bandwidth: 1Mbps, Upperlimit: 1Mbps
- qOthersHigh, CoDel, Bandwidth: 10%
- qOthersLow, CoDel, Bandwidth: 5%

Harvy66

qLink is meant to be used for non-WAN related traffic, like inter-LAN or between PFSense and the LANs. That way all of that traffic can run full LAN speed and not affect the WAN traffic coming in.

moikerz

Yes that's what I understood qLink to be for too :D But I'm not understanding the purpose of setting LAN bandwidth. Does the bandwidth of the parent queues (qLink, qInternet) need to be equal/less than the interface bandwidth?

With the config I posted, is the correct approach (see bold):

LAN bandwidth = 900Mbps
- qLink bandwidth = 896Mbps
- qInternet bandwidth = 4Mpbs
- qOthersHigh, CoDel, Bandwidth: 10%
- qOthersLow, CoDel, Bandwidth: 5%
??

It seems like that should be correct, from how I interpret what you're saying.

Harvy66

@moikerz:

But I'm not understanding the purpose of setting LAN bandwidth.

You can only shape egress traffic. This means if you want to shape your download, you need to shape it as it leaves your LAN interface, not as it comes into your WAN interface. I like to shape my download so downloads don't make my ping jump high and reduces packet-loss.

moikerz

You've inadvertently answered my question in another thread yesterday ::)

For the sake of completion for this thread, I'll link it here:
https://forum.pfsense.org/index.php?topic=112038.msg623926#msg623926
@Harvy66:

Your LAN interface is set to 1Gb/s. Your traffic is probably going into the default queue of qLink, which is limited to….. 1Gb/s. If you want your traffic to be under your qInternet, you need to place it in there somewhere

P.S. Don't place any traffic directly in qInternet, you're only supposed to place traffic in a leaf queue with HFSC.

moikerz

Here's what I've currently got:

WAN, HFSC, Bandwidth: 5Mbps
- qInternet, CoDel, Bandwidth: 5Mbps
- qNormal, Default, CoDel, Bandwidth: 10%
- qHigh, CoDel, Bandwidth: 20%
- qLow, CoDel, Bandwidth: 5%

LAN1, HFSC, Bandwidth: 900Mbps
- qLink, CoDel, Bandwidth: 895Mbps
- qInternet, CoDel, Bandwidth: 4Mbps, Upperlimit: 4Mbps
- qHigh, CoDel, Bandwidth: 20%
- qNormal, Default, CoDel, Bandwidth: 10%
- qLow, CoDel, Bandwidth: 5%

LAN2, HFSC, Bandwidth: 900Mbps
- qLink, CoDel, Bandwidth: 895Mbps
- qInternet, CoDel, Bandwidth: 1Mbps, Upperlimit: 1Mbps
- qHigh, CoDel, Bandwidth: 20%
- qNormal, Default, CoDel, Bandwidth: 10%
- qLow, CoDel, Bandwidth: 5%

And my classification rules (attached).

This give me:

5Mbps max upload
4Mbps max download for LAN1
1Mbps max download for LAN2
qHigh traffic can use 100% if available, always guaranteed 20% of parent
qNormal traffic can use 100% if available, always guaranteed 10% of parent
qLow traffic can always use 100% if available, always guaranteed 5% of parent
All traffic defaults to qNormal
email ports are low priority (qLow, saves approx 10% of bandwidth)
DNS, private cloud and OpenVPN is high priority (qHigh)
Internal-to-Internal traffic is assigned to qLink, approx 900Mbps speed

Still to do:

push Pandora, Spotify traffic into qLow
push Skype, Hangouts into qHigh
consider making the defaults qLow, and prioritize back to qNormal

floating.jpg_thumb

Harvy66

@moikerz:

consider making the defaults qLow, and prioritize back to qNormal

That's what I did.

teladero

@Harvy66:

@moikerz:

consider making the defaults qLow, and prioritize back to qNormal

That's what I did.

Can you please show us your layout? I also have two LANs that I would do this on.

Is Codel what you would use?

Harvy66

I've posted many times int he past what my setup is. I'd have to go over my message history to find it.

I use Codel as a sub-discipline and HFSC as the shaper.

KOM

It would be great if Harvy, Nullty or sideout could draft a quickie guide to pfSense HFSC and provide some basic examples for common use cases. The pfSense Book is weak on HFSC, there hasn't been a Hangout on it, and every day there is someone new trying to wrap his head around it. I'd even cough in a few bucks if it was bounty-worthy.

Nullity

@KOM:

It would be great if Harvy, Nullty or sideout could draft a quickie guide to pfSense HFSC and provide some basic examples for common use cases. The pfSense Book is weak on HFSC, there hasn't been a Hangout on it, and every day there is someone new trying to wrap his head around it. I'd even cough in a few bucks if it was bounty-worthy.

I have tried to encourage myself to do precisely that, but the combination of documentation being so unrewarding mixed with HFSC being beyond my full comprehension makes the task very daunting.

Defining "common use cases" might be a good beginning.

KOM

Common HFSC Use Cases

1 WAN / 1 LAN - [LAN: VoIP phones, ACK, DNS, WWW]
1 WAN / 2 LAN - [LAN: VoIP phones, ACK, DNS, WWW] * [DMZ: WWW, MAIL]
2 WAN / 1 LAN - [LAN: VoIP phones, ACK, DNS, WWW]
2 WAN / 2 LAN - [LAN1: VoIP phones, ACK, DNS, WWW] * [LAN2: VoIP phones, ACK, DNS, WWW]
2 WAN / 3 LAN - [LAN1: VoIP phones, ACK, DNS, WWW] * [LAN2: VoIP phones, ACK, DNS, WWW] * [DMZ: WWW, MAIL]
Per-client shaping
VPN shaping

MaxPF

@Harvy66:

I've posted many times int he past what my setup is. I'd have to go over my message history to find it.

I use Codel as a sub-discipline and HFSC as the shaper.

Do you recommend setting the priority (0-7) in the child queues? Also if using Codel as sub-discipline, do you still check the "Explicit congestion notification" option?
Thanks.