A definitive, example-driven, HFSC Reference Thread

[Harvy66]

1. You can see in your LAN traffic that all of your data is actually going into qLink, that's at least one reason why you're not actually shaping
2. did you actually set an upper limit to your qInternet? Shaping only works well if you tell it how much bandwidth you have

[georgeman]

It looks that you are not properly tagging the outbound traffic, since most of your download ends up in the default queue. Tagging is best done with floating rules, action match, interface WAN, direction OUT (I neither remember nor care about what the wizard does). Remember that floating rules processing for match rules don't stop with the first match, so the LAST matching rule wins. Make sure you catch all relevant traffic with these rules.

As regards the drops, remember that drops are NOT a bad thing. Dropping packets is a natural way TCP has to tell the other end that the packet rate needs to be lowered. It is better for it to happen on your router, where you have control of it, instead of on some upstream ISP router. This is why it is SO important to set the correct upper limits for all this to work, as Harvy66 just said. If you set a higher-than-real upper limit, your pfSense will never drop packets, they will be dropped by the ISP router instead, so you won't be actually shaping anything

[georgeman]

Also, do you have a qDefault queue on your LAN? If you don't, this is (another) flaw in the wizard. When you tag a TCP packet going out of WAN, the return traffic (the actual download) gets into the queue on LAN that has the same name as the one previously tagged on WAN. If it is not there, ends up in the default queue. This seems to be your case

[1activegeek]

Removing my message, as 1 question is irrelevant, and the other is below in next post with more detail.

[1activegeek]

Ok, so I've worked on putting together the queues, and run into an issue stopping me from being able to get very far. It seems I can't leave the Bandwidth blank on the WAN/LAN "top level" queues. So I input the 95% values in there (10/125). I then attempted putting in the numbers as advised for qInternet (95% aka 10Mb) and qLink (20%) - but I can't save and create the qLink queue. I continue to get the message:

"The sum of child bandwidth is higher than parent."

And for clarity and reference, this is the current "planned" setup. For the time being I've used 5% so I can at least build out the Queues:
(All below are Bandwidth/Linkshare m2 values made to be the same per George instructions)

WAN - 10Mb (95%)
-qInternet - 95% or 10Mb
  -qHighest - 15%
  -qACK - 20%
  -qHigh - 15%
  -qMedium - 20%
  -qDefault - 20%
  -qLow - 8%
  -qLowest - 2%
-qLink (Default) 20%
LAN - 125Mb (95%)
-qInternet - 95% or 125Mb
  -qHighest - 15%
  -qACK - 20%
  -qHigh - 15%
  -qMedium - 20%
  -qDefault - 20%
  -qLow - 8%
  -qLowest - 2%
-qLink (Default) 20%


[georgeman]

That's because the child queues are indeed exceeding the parent one!!

Just put 1Gbps or whatever the physical interface is (on the interface queue)

[Harvy66]

@georgeman:

That's because the child queues are indeed exceeding the parent one!!

Just put 1Gbps or whatever the physical interface is (on the interface queue)

If he sets the Link to 1Gb, then he needs to set the Upper Limit in the qInternet queues, which he has not.

Remember guys, "bandwidth" is the minimum bandwidth, but you need to still set your maximum.

@1activegeek:

Ok, so I've worked on putting together the queues, and run into an issue stopping me from being able to get very far. It seems I can't leave the Bandwidth blank on the WAN/LAN "top level" queues. So I input the 95% values in there (10/125). I then attempted putting in the numbers as advised for qInternet (95% aka 10Mb) and qLink (20%) - but I can't save and create the qLink queue. I continue to get the message:

"The sum of child bandwidth is higher than parent."

And for clarity and reference, this is the current "planned" setup. For the time being I've used 5% so I can at least build out the Queues:
(All below are Bandwidth/Linkshare m2 values made to be the same per George instructions)

WAN - 10Mb (95%)
-qInternet - 95% or 10Mb
  -qHighest - 15%
  -qACK - 20%
  -qHigh - 15%
  -qMedium - 20%
  -qDefault - 20%
  -qLow - 8%
  -qLowest - 2%
-qLink (Default) 20%
LAN - 125Mb (95%)
-qInternet - 95% or 125Mb
  -qHighest - 15%
  -qACK - 20%
  -qHigh - 15%
  -qMedium - 20%
  -qDefault - 20%
  -qLow - 8%
  -qLowest - 2%
-qLink (Default) 20%

95% + 20% > 100%

I don't get the LAN when you show "LAN - 125Mb (95%)" and "qInternet - 95% or 125Mb". If qInternet is 95% of LAN, and LAN is 125, then qInternet is ~119.

[Nullity]

Also, isn't "Bandwidth" really just link-share's m2 param (except on root interface queue, yeah?)? If so, that means it is only a proportional value, not a hard minimum/maximum. So, to easily avoid exceeding the parent just use low but proportionally equivilant values like 2Kb and 5Kb rather than 20Mb and 50Mb, respectively.

[1activegeek]

All - I'm going to thank everyone is advance for their help and patience. I think the picture is starting to become a bit clearer.

The reason I was setting my WAN/LAN is because pfSense won't let me NOT set these values. So I opted to fill them with the 95% of REAL bandwidth as suggested originally. This worked out to WAN = 10Mb, LAN =125Mb. The next instruction was to create the hierarchy of qInternet/qDNS-qBulk and qLink. Where qInternet was to match the 95% of REAL bandwidth (again inputting 10Mb/125Mb). qLink was then supposed to be set to 20%, which yes I get is above 100% (95+20>100). I basically hit the wall there not understanding how I could mimic the original setup.

So I think I get this but wanted to confirm a few ideas:

• WAN should be set to the actual limit (aka 95% of REAL speed) for the interface because there is no overhead available between my modem and the provider?
• qLink isn't really necessary on the WAN interface, because qLink is intended for local LAN traffic only?
• LAN should be set to the line/link speed of the interface (aka 1Gb port, set to 1Gb Bandwidth) to allow for handling LOCAL traffic as well
• UL needs to be set on qInternet for LAN (per Harvvy comment)

Assuming those 4 statements are correct, my setup would then be:

WAN - Bandwidth = 10Mb
  - qInternet - Bandwidth = 10Mb / UL = 10Mb / LS = 10Mb
    -qHighest - LS = 15%
    -qACK - LS = 20%
    -qHigh - LS = 15%
    -qMedium - LS = 20%
    -qDefault - LS = 20% (default)
    -qLow - LS = 8%
    -qLowest - LS = 2%
LAN - Bandwidth =1Gb
  - qInternet - Bandwidth = 125Mb / UL = 125Mb / LS = 125Mb
    -qHighest - LS = 15%
    -qACK - LS = 20%
    -qHigh - LS = 15%
    -qMedium - LS = 20%
    -qDefault - LS = 20%
    -qLow - LS = 8%
    -qLowest - LS = 2%
  - qLink - Bandwidth = 875Mb / UL  = 1Gb / LS = 875Mb (default)

(Crossing my fingers I'm headed in the right direction!) Thanks guys!

[georgeman]

Looks pretty much OK.

Not really a need to set UL on qLink, and I would still enforce the limit on qInternet on WAN and set the interface to the physical interface speed (so the limits are always in the same place, if you have to raise the limit in the future you will forget that it's set as well on the interface and will end up debugging this)

Tell us how it goes!

[Harvy66]

I agree with georgeman. Nothing wrong stands out.

[1activegeek]

Thanks all for the help. I've set accordingly and I think it's working as expected. Now the hard part, getting everything classed into the right queues!

[mastermindsos]

Could someone help me out please.
I followed the guide made by georgeman.
I have created these identical setup for my traffic shaper.

But I am stuck at the Floating rule.
I do not know how to create it.

I think something is wrong somewhere in the rule because when I monitored the Queue (while web browsing), the connection did not fall into the right category, instead it went into qLink which is made Default.

Also, how do I create Floating rule for the traffic shaper of LAN?
I have limited knowledge with networking and new with pfsense.
I would like to learn, please guide me. Thank you.

[georgeman]

You have set destination as "WAN net". That's just the subnet where your WAN interface resides.

What I suggested I think is to set destination "! WAN net" (not WAN net), that means every non local destination.

[mastermindsos]

Thanks for the feedback.
I do not follow you. Could you please point me?

[georgeman]

Check "invert match"

You want the rule to match everything but the WAN subnet

Or for the sake of simplicity, just select ANY. It won't make much difference. I suggest you go over this thread since the beginning if you want to understand why and how it works

[Harvy66]

I just set my floating to any/any (src/dst) and assign them directly to the WAN interface for both in/out.

[xbipin]

well i was reading this long thread, i have tried HFSC in very old versions of pfsense but then noticed it doesnt work like CBQ, basically what i wanted was suppose there is a 10MB download line and some1 is using all of it up with torrents and then a second person would like to to surf or download a single file from a site then i wanted pfsense to reduce the torrent speed and give more bandwidth to the file download as http traffic being on higher priority then p2p and when that was complete resume speed of p2p. Now i tried to achieve this using HFSC but it wasnt possible and at that time i was told HFSC is only good or comes in effect only when the internet line is saturated and doesnt kick in if it isnt saturated compared to CBQ which is active always as it borrows from other queues.

All this i had done when internet speeds were like 1Mbps or lower but now its 20Mb+ so no idea how it still works but since then im on CBQ and works well for me in situations when other higher priority queues like surfing and voip can starve p2p when they have traffic and then later give p2p full bandwidth. My aim was to only prioritize traffic rather than set a speed limit on them and CBQ works well for that but i noticed the latest pfsense doesnt have a tick box to enable borrow on a queue even though its there in the config.

let me know if HFSC can still be made to run like CBQ as i would be interested in trying if its possible

[Harvy66]

I didn't quite understand what you thought was wrong with HFSC. It pretty much guarantees a minimum amount of bandwidth and even a maximum if you want. For example. I can set my P2P to have 10% of bandwidth and everything else have 90%. When everything else is idle, P2P can use up to 100% of my bandwidth. As soon as something else wants to use bandwidth, P2P will be choked, and can be choked all the way down to 10%, which is the minimum I have given it.

I have done extreme tests where I have assigned ICMP 8kb/s on my 100Mb connection, then I saturated my connection with multiple YouTube 4K streams, some Linux ISO FTP downloads, and even some Linux ISO P2P downloads. Even with all of this going on, my ICMP traffic was unaffected. I got a near perfect ping with within less than 0.01ms difference for min/max/avg/std from when my connection was idle. If I instead moved my ICMP traffic in with the rest of all of the other traffic, suddenly I was seeing in the range of 10ms of jitter and even some packet-loss.

As far as I care, HFSC is nearly perfect at isolating traffic some each other, within reason.

Maybe the confusion is coming from a conflation of packet priority and bandwidth shaping. HFSC does not guarantee when a packet goes out, it only guarantees an upper-bound. CBQ actually round-robins the queues, processing higher bandwidth queues first. This means higher bandwidth queues get lower pings, HSFC does not have this issue. HFSC instead dynamically calculates the "priority" of a queue based on its historical bandwidth usages and makes sure all queues gets their assigned share. The cool thing about HFSC is it is so good at this, if I set a queue to have a minimum and a maximum that are the same, that queue will be virtually as good as dedicated bandwidth and will never know other queues exist. When I do this, I am seeing identical ping statistics to withing 0.01ms accuracy, which is the precision limit of my ping tool. As far as I care, that's perfect. At this point the bottleneck of scheduling is less an issue of the shaper and more an issue of the OS Kernel handling interrupts.

One thing to be aware of is HFSC effectively splits bandwidth in ratios. If you assign one queue 1% and another queue 1%, and these are the only two queues doing anything, there will effectively be a 50/50 sharing. Where this can break expectations is how things ramp up. When my line is idle, a single high bandwidth stream can go from 0 to 100Mb is a fraction of a second, with a nearly vertical line on my bandwidth graph. But if another queue is already saturating the link, and even if that other queue is assigned only 20% bandwidth and my measured queue 80%, HFSC won't suddenly starve the "lower priority" queue. Instead of going from 0 to 100% in less than a second, it may take 1-2 seconds, quickly getting to  half of its guaranteed fair share, but then having more of a mild arc than a strait line taking back the later half. My assumption with CBQ being a strict queue, it would just cut off all "borrowed" bandwidth immediately, which could lead to high packet-loss, bufferbloat, and massive jitter for the lesser queue. None of which is desirable. What's the point of borrowing bandwidth if it actually makes your experience worse?

[xbipin]

Wow, that was a very good explanation, thanks for that. Well I don't criticize the use of hfsc, it might better than the rest but in the olden times when i had a internet connection of 1mbps max, I had tried a lot to configure it such that when anyone wants to surf or make voip calls, they wouldn't be affected by p2p downloads as that would slow down but at that times there was ADSL here and to know the actual up and down limit of the line was a mystery because ADSL interference would affect everything and in this case hfsc would basically almost not kick in unless the line was saturated but due to the line speed fluctuating it would never seems to come in effect because settings a download speed limit of 1mbps or a little less would most of the times never reach due to the isp connection being so crappy, I was so frustrated that ermal who I guess who wrote the traffic shaping php code also mentioned this that hfsc wouldn't help me because it would never kick in unless the line is very busy or saturated. He recommended using cbq which worked flawless for me and still does but I still want to switch to hfsc but the issue is with so many parameters it sometimes gets very tricky to configure the way u want it and that's why most basic users would run away from it.

The main problem i faced with HSFC was if line was saturated and voip call initiated, then even though voip had higher priority it would still suffer the audio breakup even after a few seconds into the call, probably like u said HFSC doesnt gaurantee which packets go out first but as soon as i went to CBQ, this almost disappeared.

CBQ though has one bug which doesnt allow a 3 tier shaper so u need to stick to 2, i had filed a bug ages ago and it was simply closed and not fixed
https://redmine.pfsense.org/issues/1053

Coming back to HFSC, i really want to switch to it after u mentioned u have tested it thoroughly, my current CBQ queues r as below

WAN - CBQ - 3Mb
– qACK - priority 6 - bandwidth 30% - borrow on - codel on
-- qOthersDefault - priority 4 - bandwidth 7% - borrow on - codel on
-- qP2P - priority 1 - bandwidth 5% - borrow on - codel on
-- qVoIP - priority 7 - bandwidth 48% - borrow on
-- qOthersHigh - priority 5 - bandwidth 10% - borrow on - codel on

WAN - CBQ - 12Mb
-- qACK - priority 6 - bandwidth 15% - borrow on - codel on
-- qOthersDefault - priority 4 - bandwidth 45% - borrow on - codel on
-- qP2P - priority 1 - bandwidth 5% - borrow on - codel on
-- qVoIP - priority 7 - bandwidth 25% - borrow on
-- qOthersHigh - priority 5 - bandwidth 10% - borrow on - codel on

now using this and the bug i mentioned earlier im not able to create a 3 tier shaper and put traffic to pfsense GUI outside these queues which r mainly used for internet shaping so suppose i want to connect to pfsense or upload or download any files from it then im being shaped always so what i want to do is convert this CBQ queues to HFSC and add that extra tier so access to pfsense gui isnt slowed down.

I would really appreciate if u can help me convert this to HFSC queues, keeping similar config intact so probably i can test that out and report my findings after having left HFSC for so long now

edit
qP2P is the default queue so whats not put in other queues goes to qp2p

in my case the download speed being 12mbps that doesnt create a bottleneck but its the upload speed which is 3mbps is where the bottleneck happens so currently i use CBQ and limiters to limit the upload speed