Yes SamTzu is the same issue and here we don't run Squid…And i think is a generic issue but i believe still not fixed

LOL  ;) I should have mention it, I upgraded to 2.2.1 after my first post, delete the existing traffic shaper rules, and re created them

@Nullity:

@Harvy66:

Don't forget to set the bandwidth on the interface. You should set it to a value slightly less than your actual upload speed. If you get 5Mb to say speedtest, then set your  upload to 4.9Mb.

It also depends on the stability of your upload. If it fluctuates a lot, you may want to lower if further. In theory, you should set your interface to be less than your lowest speed, but that is not always practical.

The emboldened section of the quote is very important. If pfSense is not the "bottle-neck" of throughput, then traffic-shaping does not work. You might try setting qACK to ~500Kb and set other (unimportant) queues to 1Kb. Leave m1/d/m2 alone, for now.
Perhaps try check-marking the "Codel Active Queue" box.

Regardless of the algorithm you choose (HFSC, CBQ, PRIQ, etc), you need to follow standard QoS practices and understand a good amount of networking basics. I quickly found out that "easy" and "QoS" rarely coincide, sadly.

<- What he said  ;D

There is no easy money in this… but this golden link helped me with basic understanding of QoS:

http://www.linksysinfo.org/index.php?threads/using-qos-tutorial-and-discussion.28349/

If you read all of it... you can go further.
I also thought that using basic pFSense wizard would get me covered but...

State table uses 10% of RAM by default, but you can modify this as per SamTzu.

https://doc.pfsense.org/index.php/How_can_I_increase_the_state_table_size

Thanks for the reply guys.

Tried simple limiters?

Drops are a good thing to signal the sender to back off, but drops are not good if they're too aggressive. Codel just makes it simple. It can be useful on the LAN side, but less useful. Especially since download tends to be much faster than upload for most people.

Shaping downloads as worked decently well for me, but not nearly as well as upload. Upload is a 1Gb link going into a 100Mb link, so going over 100Mb is not an issue because the shaper will buffer the data and not affect the other queues. The problem with download is it's a 100Mb link going into a 1Gb link, so I need to make sure my download never goes above 100Mb because it will buffer upstream instead of in my traffic shaper.

For the most part this just means I can't use a tight 98% of my link speed for download, it needs to be a bit looser, like 95%, which wastes more bandwidth. One issue that I have found out is because PFSense is stateful, when the WAN interface see duplicate TCP packets, it just drops the packet and sends a dup-ack. Since this Dup packet never makes it to the LAN interface, the LAN thinks there is less than 100Mb coming in, so it doesn't cause the other traffic to back-off. This is not an "issue" with PFSense, but an issue with bad-actors.

Well, I just tested my theory and it failed. HFSC needs to be configured as the throughput bottle-neck with the root queue's "Bandwidth" parameter.

I will be testing more scenerios, like perhaps moving the actual link's queue from the root (root queue cannot change it's bandwidth dynamically, apparently) to an interior leaf queue (a leaf's bandwidth is dynamic). I will report back if find anything new. How the hierarchy of the queues is laid out is actually important, so I have some hope it may work. :)

It may be interesting to note that HFSC was not originally implemented with an "upper-limit".
https://github.com/freebsd/freebsd/blob/428b45aa532260e8c6ddf0217ec31db2234d29a8/sys/contrib/altq/altq/altq_hfsc.c#L39

Anyway, here are my results:

WAN Bandwidth set to 640Kb. (Less than real-world throughput of 666Kb)

pfTop: Up Queue 1-6/6, View: queue, Cache: 10000                        16:55:57 QUEUE              BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP P/S  B/S root_pppoe0      640K hfsc  0    0    0      0      0    0            0    0 qBulk            1000 hfsc      214 76697      0      0    1            1  695 qACK            500K hfsc    1697 94153      0      0    0            3  211 qNTP            25000 hfsc        0    0      0      0    0            0    0 qTEST1          **1000** hfsc    1285 1500K    19  7521  38            9  **11K** qTEST2          **6000** hfsc    3067 3792K      0      0  35            52  **67K**

You can see that the ratios between the 2 test queues are almost exactly 1:6.
These are the speeds recorded in my ftp client, "lftp", during the above test. The speeds are not pasted exactly as I observed, because I had to freeze the terminal with Ctrl-S.
tmp/linux-3.18.5.tar.xz.1' at 5651256 (11%) **63.2K/s** eta:11m [Sending data] progit.en.pdf' at 3180520 (56%) 11.0K/s eta:2m [Sending data]

WAN Bandwidth set to 250Mb.

pfTop: Up Queue 1-6/6, View: queue, Cache: 10000                        17:01:25 QUEUE              BW SCH  PR  PKTS BYTES DROP_P DROP_B QLEN BORR SUSP P/S  B/S root_pppoe0      250M hfsc  0    0    0      0      0    0            0    0 qBulk            1000 hfsc      14  2667      0      0    0            0    0 qACK            500K hfsc      34  1874      0      0    0            0    0 qNTP            25000 hfsc        0    0      0      0    0            0    0 qTEST1          **1000** hfsc    1592 1921K      0      0    0            27  **34K** qTEST2          **6000** hfsc    1551 1940K      0      0    0            35  **46K**

As you can see, this test did not function as I expected.

Edit: "Bandwidth" and link-share's "m2" are set to the same value.

You can set up shaping for CP in the CP configuration. As for the rest, absolutely no information here.

@Harvy66:

I'm not disagreeing with them, I'm just saying math proofs don't tell you how to use HFSC practically, only that it works "as expected" assuming you understand it. 99% of well experienced and educated people in networking will not understand how network traffic works in sub 1ms time scales, so don't even think about time scales that small. I do agree that my "evidence" is anecdotal, I have limited tools. Another term for "anecdotes" is "data points",  albeit with high uncertainty, there is some truth to anecdotes.

In order to property configure HFSC the way you you've been giving examples, you would need to know these and how these interact:

Packet Size PPS(Packets per Second) during transmission Average Bandwidth Number of flows Distribution of packets within a flow Distribution of packets among all flows

The examples the HFSC links use are simple to prove a point, but do not reflect the complications of a real network. To take the example at face value is to over-simplify the issue, in other words, don't think about individual packets unless that's exactly what happens at the bandwidths you're working with.

**Up to this point I've been arguing an almost laissez faire sort of configuration while you've been arguing an extremely precise. I know I was mostly playing devil's advocate to give an alternative view, but I think the middle ground is best. In this paper(http://www.cs.cmu.edu/~hzhang/papers/SIGCOM97.pdf) that you linked at one point in another thread, it talks about 160byte VoIP packets with an average of 64Kb/s. The way they configured and talked about the burst duration is as a target latency. They set the duration to 5ms as they wanted a 5ms target. m1 was set to the packet size, 160 bytes(not bandwidth like you've mentioned), then spread over the 5ms, they gave the example of 256Kb/s. Because 160bytes/5ms=256Kb.

What I was able to gain from this example is you set the m1 bandwidth equal to the size you wish to "burst", and the duration to the time in which you wish the extra bytes to be transferred.

I feel fairly confident that the d(duration), for the purpose of latency sensitive traffic, should be set to your target worst latency. m1 should be thought of not as bandwidth, but the size in bits of the total number of packets relieved during that duration. m2 would be set as the average amount of bandwidth consumed.**

Incorrect. m1 and m2 define bandwidth.

@Nullity:

This quote from the HFSC paper is useful because it clarifies differences between the parameters used in the paper (u-max, d-max, r), and the parameters that pfSense uses (m1, d, m2).

Each session i is characterized by three parameters: the largest unit of work, denoted u-max, for which the session requires delay guarantee, the guaranteed delay d-max, and the session's average rate r. As an example, if a session requires per packet delay guarantee, then u-max represents the maximum size of a packet. Similarly, a video or an audio session can require per frame delay guarantee, by setting u-max to the maximum size of a frame. The session's requirements are mapped onto a two-piece linear service curve, which for computation efficiency is defined by the following three parameters: the slope of the first segment m1, the slope of the second segment m2, and the x-coordinate of the intersection between the two segments x. The mapping (u-max, d-max, r) -> (m1, x, m2) for both concave and convex curves is illustrated in Figure 8.

The paper uses u-max (packet/frame size) in it's examples, but pfSense uses m1 (bandwidth).
When glancing at a configuration using m1, d, and m2, it is obvious whether the parameters are meant to decrease delay (m1 > m2) or increase delay (m1 < m2).
The u-max/d-max notation is not as intuitive.

I pulled the quote from the HFSC thread, if you would like to move this HFSC-related conversation there.
https://forum.pfsense.org/index.php?topic=89367.0

Edit: Fixed typo in quote from HFSC paper.

You can create a time schedule in pfsense in  –> firewall --> schedules then apply this schedule to some rule.

This rule will be disabled if out of time range (8am 9am)

Ok, but in earlier versions that doesn't happen

Ahh, thanks, I had been under the impression Limiters applied globally, but I missed the In/Out options when I was looking at it.

Regards,
Rob.

Create a queue on the WAN you want to limit.  Put an upperlimit of 5Mbit/s  Call it qWANLimit or something
Create a queue on LAN with an upperlimit of 50Mbit/s  Name it the same thing.

Create a floating rule on the WAN interface you want to limit, direction out, match all traffic, and set the queue to qWANLimit.

If you have inbound NAT translations or connections, you have to set the queues on those rules too.

If you can, test your queues with upperlimits of, say, 10% of your capacity so you know they're working, then bump them back up.

Sorry, but my test network is on Xen and I haven't rebuilt it with 2.1.5 so I have no altq to test it with.

This is also imperfect in that you should separate TCP from UDP and set an ACK queue on TCP, etc.  This is just a general idea of how to get traffic for one WAN into the proper queue.

Layer7 is broke in the latest release afaik.