Traffic Shaping broke my LAN - topic has deviated



  • I made some changes to the HFSC settings for my LAN interface and my Internet dropped out and I could no longer ping PFSense. I had to do a forceful reboot. It came back up and was pingable for a little bit, then died.

    I rebooted yet again. Came back up for a little bit, long enough for me to log in and I saw GEOM was degraded. At this point I had my keyboard and monitor plugged in so I did a graceful shutdown.

    I unplugged the other SSD. Came back up, lasted for a bit, went back down.

    At the console I tried pinging the Internet, worked just fine. I tried pinging my computer on the LAN, time out. I rebooted once more and immediately went into traffic shapping and removed all shaping.. BAM! LAN works again!

    At the console I restored to a prior config which had most of my shaping rules. I am now back up and running. I'll have to re attach my other SSD. An annoyance about GEOM is it reimaged the entire other SSD, which dropped my wear level by 1. Kind of sucks to write 120GB of data when only a few KB at most are different. ZFS?

    P.S. This issue made it so all VLANs on the LAN interface no longer worked, it did not just affect my LAN VLAN, but also my Admin VLAN. The entire port was affected.

    ewwwwwww



  • I reattached my other SSD, is still rebuilding, attempted to view RRD graphs and the web interface stopped working. HD activity light is still solid and other than a brief bunch of time outs, the box started to respond to pings again and the Internet started working again. Sucks that the web service no longer works for now. Hopefully it'll fix itself, I hate doing forceful reboots because it seems to write another 120GB to my SSD and wears it out much faster.

    edit: It's been many minutes now. HD activity light finally turned off, but the web interface is still not working. Internet seems to be in working order. I'm going to plug in my keyboard and blindly type and reboot. The monitor doesn't work if it was not plugged in when the box booted. Even worse, when I plugged in my monitor, my internet stopped working. At least it rebooted gracefully.



  • It seemed to work until I tried poking around in PFSense's web interface, then things would break, no response on the LAN port. I finally logged in after a reboot and removed all traffic shaping on the LAN, and I can now view the RRD without crashing my LAN port. Dear lord!

    edit: I placed just a default queue and enabled codel for the time being for my LAN. My WAN's shaping seems to be just fine.


  • Banned

    @Harvy66:

    It seemed to work until I tried poking around in PFSense's web interface, then things would break, no response on the LAN port

    What kind of poking? Are you repeatedly adding and removing shapers?



  • Just changing pages to other than the dashboard. RRD, System Logs, Traffic Shaper, etc. I was not making any changes during this time.

    edit: Clicking on the RRD was a near instant loss of connection, but going to the Traffic Shaping gave me a few seconds before completely dying. I used this small window later on to remove all shaping on the LAN interface.


  • LAYER 8 Netgate

    I would take a copy of the current config then roll back to the backup/snapshot I took before messing with my shaper settings.

    Then maybe diff the working config with the new-and-improved config to see where I screwed it up.

    But it sounds like you might have some other coincidental failure of some kind.



  • The only time the traffic-shaper has broke my LAN is when I would forget to change the LAN queue bandwidth limit from (k?)bit/sec to mbit/gbit… self-inflicted DoS. :)



  • @Nullity:

    The only time the traffic-shaper has broke my LAN is when I would forget to change the LAN queue bandwidth limit from (k?)bit/sec to mbit/gbit… self-inflicted DoS. :)

    For me, it actually affected other VLANs on the same physical interface.



  • I just had it happen again when I tried this setup

    LAN - HFSC 95Mbit
    –pInternet: Upper Limit 95Mb
    ----qHigh: LS: 49.42% 5ms 30.55%  Queue 4069+Codel
    ----qNormal: LS: 30.55%  Queue 4069+Codel
    ----qDefault: LS: 15.27%  Queue 4069+Codel
    ----qACK: LS 0.01% RT 20%

    After I hit save, I lost LAN access.

    This is my current setup and it works just fine
    LAN - HFSC 95Mbit
    --qHigh: LS: 49.42% 5ms 30.55%  Queue 4069+Codel
    --qNormal: LS: 30.55%  Queue 4069+Codel
    --qDefault: LS: 15.27%  Queue 4069+Codel
    --qACK: LS 0.01% RT 20% Queue: 4096 FIFO

    I would like to do more testing, but my wife has hit her limit.

    My next plan is to backup my config and do a clean install, some time after 2.2.1. I also plan to repartition my SSDs to only be 20GB in size instead of the full 120GB, that way GEOM doesn't re-write the full 120GB every time it gets mad.

    In case someone is wondering, I'm using the 80:20 rule, such that High and normal are 40% each and default is the 20%. But then I scaled them down so qHigh can have a 1.618x burst for 5ms, golden ratio, which is about 11,000 bytes worth. Plenty for a short burst of small packets.



  • The only strange thing I see is the overly precise percentages.
    Maybe try whole numbers for the percentages?



  • @Harvy66:

    In case someone is wondering, I'm using the 80:20 rule, such that High and normal are 40% each and default is the 20%. But then I scaled them down so qHigh can have a 1.618x burst for 5ms, golden ratio, which is about 11,000 bytes worth. Plenty for a short burst of small packets.

    If you are only using HFSC for it's limiting properties (since it cannot prioritize incoming packets that have already arrived), why not switch to limiters?

    The burst for qHigh is unneeded. You will only see imperceptible, sub-millisecond improvements. (~50Mbit vs ~30Mbit)
    Isn't a 5ms burst of ~50Mbit equal to ~31,250 bytes, not 11,000 bytes?

    Edit: I think the burst is copied to every stream in the queue (guaranteeing each stream's delay), but the m2 is applied to the entire queues average bandwidth. This is the only way it makes sense to me. If the m1/d params were applied to the entire queues delay, then delay would be dependant on the number of streams, which would preclude any guarantee of delay.

    I mention that because m1 and d should never be calculated to be larger than MTU, according to my previous paragraph. I am still reading the referenced papers in the HFSC paper to find a definitive answer.



  • I updated my posts, those percentages were in reference to linkshare, only the one upper bound is used by the parent queue. My plan was to make the root 1Gb again.

    "since it cannot prioritize incoming packets that have already arrived" - Actually, the only packets you can prioritize are ones that have already arrived. I want HFSC to artificially delay my bursty YouTube and Netflix streams and let my High priority stuff through.

    The 5ms burst would be on the difference between the nominal and burst, so (0.4942-0.3055)95,000,0000.005/8=11,204.0625 bytes, which is almost 20 of the average packet size(576bytes). HFSC does everything on a curve, so I'm not sure if burst exactly works that way, but I figure it's close.



  • @Harvy66:

    "since it cannot prioritize incoming packets that have already arrived" - Actually, the only packets you can prioritize are ones that have already arrived. I want HFSC to artificially delay my bursty YouTube and Netflix streams and let my High priority stuff through.

    ~~Incoming WAN traffic will be shaped when leaving the LAN. A 100Mbit WAN link cannot saturate your 1Gbit LAN. Since QoS only serves a purpose on a congested link, QoS is useless on your LAN for incoming WAN traffic.

    The only thing HFSC is accomplishing is rate limiting.

    You could create limiters and allocate 40/40/20 of your incoming bandwidth and it would accomplish practically the same thing.~~

    Edit: Disregard all this. I do not fully understand how incoming traffic-shaping works.



  • @Harvy66:

    The 5ms burst would be on the difference between the nominal and burst, so (0.4942-0.3055)95,000,0000.005/8=11,204.0625 bytes, which is almost 20 of the average packet size(576bytes). HFSC does everything on a curve, so I'm not sure if burst exactly works that way, but I figure it's close.

    HFSC's m1 and d parameters achieve the same goal as the d-max and u-max parameters, which are the parameters used in the HFSC paper. The u-max parameter is the largest unit of work (aka packet size) and d-max is the guaranteed delay (aka milliseconds to fully transmit).

    The paper mentions an example for an audio stream with 160byte packets;
    u-max=160 bytes
    d-max=5 ms
    r=64 Kbps (r is m2)

    Trasnslating this to m1/d/m2 notation is; (160bytes in 5ms is 256Kbps)
    m1=256 Kbps
    d=5 ms
    m2=64 Kbps

    My point is, m1/d seems to be single packet-size only, not multiple. Bursting 30Kbyte seems unneeded.
    Be cautious with HFSC. Misconfigurations are rampant.



  • @Nullity:

    @Harvy66:

    "since it cannot prioritize incoming packets that have already arrived" - Actually, the only packets you can prioritize are ones that have already arrived. I want HFSC to artificially delay my bursty YouTube and Netflix streams and let my High priority stuff through.

    Incoming WAN traffic will be shaped when leaving the LAN. A 100Mbit WAN link cannot saturate your 1Gbit LAN. Since QoS only serves a purpose on a congested link, QoS is useless on your LAN for incoming WAN traffic.

    The only thing HFSC is accomplishing is rate limiting.

    You could create limiters and allocate 40/40/20 of your incoming bandwidth and it would accomplish practically the same thing.

    My ISP actually bursts 1Gb/s for tens of milliseconds regularly. This was more of an issue before my ISP did AQM, but it can cause packet-loss issues if I let the burst through. Google sets the receive window too large if I let the burst through at full rate.

    @Nullity:

    @Harvy66:

    The 5ms burst would be on the difference between the nominal and burst, so (0.4942-0.3055)95,000,0000.005/8=11,204.0625 bytes, which is almost 20 of the average packet size(576bytes). HFSC does everything on a curve, so I'm not sure if burst exactly works that way, but I figure it's close.

    HFSC's m1 and d parameters achieve the same goal as the d-max and u-max parameters, which are the parameters used in the HFSC paper. The u-max parameter is the largest unit of work (aka packet size) and d-max is the guaranteed delay (aka milliseconds to fully transmit).

    The paper mentions an example for an audio stream with 160byte packets;
    u-max=160 bytes
    d-max=5 ms
    r=64 Kbps (r is m2)

    Trasnslating this to m1/d/m2 notation is; (160bytes in 5ms is 256Kbps)
    m1=256 Kbps
    d=5 ms
    m2=64 Kbps

    My point is, m1/d seems to be single packet-size only, not multiple. Bursting 30Kbyte seems unneeded.
    Be cautious with HFSC. Misconfigurations are rampant.

    You can also use the burst to mimic speedboost. My old ISP used to give a 100Mb "boost" for about 8 seconds, then return back to 30Mb. That would be similar to 100Mb 8000ms 30Mb HFSC setting.

    A "burst" is really just a temporary change in ratios. It is more useful for smaller that tend to have strong bursting characteristics.



  • @Harvy66:

    You can also use the burst to mimic speedboost. My old ISP used to give a 100Mb "boost" for about 8 seconds, then return back to 30Mb. That would be similar to 100Mb 8000ms 30Mb HFSC setting.

    A "burst" is really just a temporary change in ratios. It is more useful for smaller that tend to have strong bursting characteristics.

    The HFSC paper never directly mentions using m1/d this way. Have you verified that HFSC can be used this way?

    It might be a better idea to use upper-limit to achieve a "speedboost". (Though, remember, HFSC created the 2-part service-curve to explicitly decouple bandwidth and delay, not to achieve a "speedboost". Your 8second speedboost might be possible, but it may also be a misuse that will cause unforeseen problems.)

    My criticism is an attempt to make sure you are configuring HFSC cautiously and avoiding potentially breaking HFSC/traffic-shaping by misconfiguring it.



  • M1 is a bandwidth setting, exactly the same as M2. Sum(max(m1,m2)) cannot be greater than 100% of the parent queue.



  • @Harvy66:

    M1 is a bandwidth setting, exactly the same as M2. Sum(max(m1,m2)) cannot be greater than 100% of the parent queue.

    Yeah, they are both bandwidths, but m1 and d define delay while m2 defines average bitrate. Perhaps they can be used in other ways, but that may cause undefined results. I would configure HFSC conservatively and not make guesses.

    Since HFSC is primarily meant to guarantee per-packet delay, the calculated byte-size of "m1 × (d ÷ 1000) × 8" should not greatly exceed the interface's MTU. (I hope I got that math right.)

    I am still unclear how m1/d interact with link-share, since link-share makes no guarantees about bandwidth or delay. Link I mentioned earlier, if you must attempt the "speedboost", maybe use upper-limit by setting m1=50Mb, d=5, and m2=30Mb.

    Maybe switch to a more conservative configuration and try just setting m2 and see if it fixes your broken LAN?



  • I would like to do more testing, but whit one machine, I can't afford the wife to get angry. I'm leaving it a lone for now.

    I try not to care what HFSC does to individual packets, I only really care what it does on average. If I set a burst from 30% to 50% for 5ms, to me that just means for 5ms, the ratio of bandwidth between queues will be different. bursts are just ways to temporarily modify bandwidth with little change to the overall average.



  • @Harvy66:

    I would like to do more testing, but whit one machine, I can't afford the wife to get angry. I'm leaving it a lone for now.

    I try not to care what HFSC does to individual packets, I only really care what it does on average. If I set a burst from 30% to 50% for 5ms, to me that just means for 5ms, the ratio of bandwidth between queues will be different. bursts are just ways to temporarily modify bandwidth with little change to the overall average.

    Yeah… I use my internet gateway for testing purposes too. :(

    You may be right about using bursts as speedboosts (increased average bitrate for all packets transmitted during a 5ms time-span, for example), but since the HFSC paper never explicitly uses it like that (m1/d is only used to modify delay, while m2 is the average), I think we need to do some testing to prove/disprove your theory.

    It would be great if HFSC's m1/d could be used to modify the initial average bitrate of a connection/session, but I am obviously skeptical.

    A quick thought experiment... your theory seems impossible for UDP since all individual UDP packets are their own individual transmission sessions (connectionless). If you employed m1/d like in your speedboost thoery, all UDP packets could send at 50Mbit for the first 5ms but must stay below a 30Mbit average. Since each individual UDP packet is a new session, and each new session gets an initial 50Mbit "speedboost" for 5ms, your average UDP bitrate could reach 50Mbit, because all UDP packets are transmitted during the "speedboost". The average bitrate could forever be above the m2 configured average bitrate of 30Mbit. Your "speedboost" theory breaks HFSC by causing m2 to be non-functioning.

    The only way m1 and d function without breaking UDP sessions with HFSC is if m1 and d only modify individual packet transmission delay, and not the initial average session bitrate like in your "speedboost" theory.
    I am pretty sure this applies to TCP as well, but I am not sure. The HFSC paper focuses on generic packet delay, never mentioning "TCP" and only mentions "UDP" once during a simulation of link-share's bandwidth sharing capabilities.

    It seems like HFSC's m1/d params only control the transmission delay of generic packets/frames, and nothing else.

    Any criticisms are welcome. These conversations help me understand HFSC.



  • UDP " stateless" as in the state must be handled by the application, not the protocol, but even PFSense has a notion of a UDP "State". UDP doesn't "burst". The most common usage of UDP is a flow of packets that are sent at standard intervals or event based. Most UDP flows do not "burst" in the sense of TCP trying to send data, UDP just blindly sends data with no concern for congestion.

    When UDP "bursts", it does so only in the sense that many packets so happened to arrived at the "same time", in some sense of the term "same time'.

    And HFSC doesn't burst based on new sessions, it bursts based on the current usage of the queue. If a queue has 30Mb allocated, but only 15Mb is being used, but suddenly a bunch of traffic arrives "at the same time", HFSC will temporarily allow a higher amount of bandwidth through that is above the 30Mb. I think the burst is inverse related to the current usage of the queue, such that if the queue is at 100% usage, the burst will be 0% of the burst amount, but if you're at 50% usage, the amount of burst will be 50% of the assigned amount. Without knowing the exactness of how the burst works, I cannot know how it will function at the micro time scale.

    The best way to think about HFSC is not in bandwidth, but in ratios. Actual packet scheduling is based on the current ratios and packet ordering is not part of the algorithm. HFSC decouples bandwidth and latency such that if all queues are at 100% utilization, regardless of the amount of bandwidth assigned, they will all experience the same latency. While latency for all queues are equal, the ordering of the packets are not guaranteed, unless you use the priority option, but from the sounds of it, it makes little difference.


  • LAYER 8 Netgate

    Connectionless != Stateless.



  • I read a little about burst and it seems that it's more "bucket" like in that "credit" for the burst accumulates with some relation of the lack of usage of the entire allocated bandwidth. So 50% usage would not mean you can only use 50% of the burst, but the rate at which credit for the burst accumulate may be reduced by 50%. This rate may or may not be linear and in direction relation to the volume of the burst.

    So I don't know if this example would be correct: If you have a 10Mb burst for 10ms, once you've used the burst, you need to not use 10Mb for another 10ms in order to get the full burst again.

    It does sound more like you save current bandwidth now in order to burst bandwidth later, and not use a burst now but get penalized in the future. My guess is it's volume based and over a smooth curve, as HFSC tends to have a smoothing effect. So my current guess is if you have a 10Mb burst for 10ms, but you "saved" 5Mb of credit since the last burst, you will get that 5Mb spread smoothly over that same 10ms burst.

    Of the information that I have seen, "burst" just changes the allocated bandwidth, but does not affect when a packet leaves. It sounds the same as temporarily changing the m2 to the m1 for a duration.



  • @Harvy66:

    UDP " stateless" as in the state must be handled by the application, not the protocol, but even PFSense has a notion of a UDP "State". UDP doesn't "burst". The most common usage of UDP is a flow of packets that are sent at standard intervals or event based. Most UDP flows do not "burst" in the sense of TCP trying to send data, UDP just blindly sends data with no concern for congestion.

    When UDP "bursts", it does so only in the sense that many packets so happened to arrived at the "same time", in some sense of the term "same time'.

    And HFSC doesn't burst based on new sessions, it bursts based on the current usage of the queue. If a queue has 30Mb allocated, but only 15Mb is being used, but suddenly a bunch of traffic arrives "at the same time", HFSC will temporarily allow a higher amount of bandwidth through that is above the 30Mb. I think the burst is inverse related to the current usage of the queue, such that if the queue is at 100% usage, the burst will be 0% of the burst amount, but if you're at 50% usage, the amount of burst will be 50% of the assigned amount. Without knowing the exactness of how the burst works, I cannot know how it will function at the micro time scale.

    The best way to think about HFSC is not in bandwidth, but in ratios. Actual packet scheduling is based on the current ratios and packet ordering is not part of the algorithm. HFSC decouples bandwidth and latency such that if all queues are at 100% utilization, regardless of the amount of bandwidth assigned, they will all experience the same latency. While latency for all queues are equal, the ordering of the packets are not guaranteed, unless you use the priority option, but from the sounds of it, it makes little difference.

    I think it is important to know that the HFSC authors never referred to m1/d as a "burst". They only called it one thing, a way of creating a 2-part service curve to decouple bandwidth and delay allocation. I am not sure where the confusing "burst" explanation came from.

    Here is a quote about priority from the first sentence of the paper:

    In this paper, we study hierarchical resource management models and algorithms that support both link-sharing and guaranteed realtime services with priority (decoupled delay and bandwidth allocation).

    They say that because m1/d control delay (not "speedboost"), which decides the ordering of the packets. The service curve with earliest deadline (lowest delay, without getting overly technical) will be sent first, as mentioned below.

    HFSC does have an algorithm for packet ordering, it is based on SCED (Service Curve Earliest Deadline first). See the following paper for it's origins: http://www-afs.secure-endpoints.com/afs/ece/u/anoopr/OldFiles/742/Scheduling for quality of service guarantees via service curves.pdf
    I think HFSC modifed it to support both real and virtual times, but I am not completely sure… still trying to absorb a couple dozen academic papers.  :-\

    Only link-share works with ratios, because it uses Virtual Time. Real-time, obviously, works with Real Time and that is why it can guarantee delay and bandwidth when link-share does not.



  • @Harvy66:

    I read a little about burst and it seems that it's more "bucket" like in that "credit" for the burst accumulates with some relation of the lack of usage of the entire allocated bandwidth. So 50% usage would not mean you can only use 50% of the burst, but the rate at which credit for the burst accumulate may be reduced by 50%. This rate may or may not be linear and in direction relation to the volume of the burst.

    So I don't know if this example would be correct: If you have a 10Mb burst for 10ms, once you've used the burst, you need to not use 10Mb for another 10ms in order to get the full burst again.

    It does sound more like you save current bandwidth now in order to burst bandwidth later, and not use a burst now but get penalized in the future. My guess is it's volume based and over a smooth curve, as HFSC tends to have a smoothing effect. So my current guess is if you have a 10Mb burst for 10ms, but you "saved" 5Mb of credit since the last burst, you will get that 5Mb spread smoothly over that same 10ms burst.

    Of the information that I have seen, "burst" just changes the allocated bandwidth, but does not affect when a packet leaves. It sounds the same as temporarily changing the m2 to the m1 for a duration.

    "Burst" is a misnomer. m1/d only control a packet's delay. A token bucket is something different which can actually achieve a "burst" as you perceive it.

    Expanding on your HFSC example, if
    m1=10Mb
    d=10ms
    m2=5Mb

    Then we could translate this to u-max/d-max
    u-max=12500 Bytes [ m1×(d÷1000)÷8 = u-max ]
    d-max=10ms
    r=5Mb (aka m2)

    This means a packet/frame with the maximum size of 12000Bytes (wayyy oversized) is guaranteed to be transmitted within 10ms, but the average bitrate cannot go over 5Mb. Once the packet is "burst", no other packets can be sent until the long-term average has fallen low-enough that another u-max sized packet can be transmitted at "burst" speeds without raising the long-term average above the m1 bitrate. (I am unsure how long "long-term" is)

    That is how the HFSC paper explains it.

    We should probably stop calling m1/d a "burst" because it is, at best, an over-simplification, and more likely plain wrong because m1 can be set to 0. If m1=0, there is no "burst", only added delay.



  • I've read similar examples about burst. So it sounds like "bursting" allows you to acquire a certain amount of "bandwidth debt", up to the amount set in the burst. It is kind of an extra examples to use a 1500byte packets on a 400kb link, but this hypothetical situation gets used many times. So in this case, the 1500byte packet cannot be dequeued until enough "bandwidth credits" have been accumulated, but a burst value gets set which allows the queue to acquire some "bandwidth debt", allowing it to dequeue much sooner.

    This situation is a bit unnatural, except in the case of people with slow links, because the MTU is extremely large relative to the bandwidth, plus they're using a 1500byte packet and claiming it to be VoIP. 1500bytes on a 400Kb link is 30ms and that's ignoring how much bandwidth the example queue is given. In a more real world example, you have a 1Gb link and 200 byte VoIP packets, which gives you 0.0016ms. Even if you set aside 10Mb for the VoIP queue, you're still talking about an average 0.16ms queue time at saturation.

    My argument is that while thinking about individual large packets on slow links is important for showing how the algorithm works, in the real world, you're more likely to have to think about statistical averages when configuring burst values. If I had a 10Gb link and I needed to set aside 2Gb for VoIP, I won't be trying to figure by burst value for a 0.0008ms duration for a 200 byte packet, I'll be more concerned about what is going on at a more macro level, relative to an 80ns packet delay, in the 1ms+ range. What is the threshold for VoIP delay before it becomes perceptible? 5ms? 10ms? 20ms? Those are the bursts I would be focusing on or some fraction of them. Maybe the optimal burst length is 50% of your target delay. If I want VoIP to get dequeued before a 10ms delay, maybe I want a 5ms burst target.

    In the end, it all comes down to bandwidth management. Assuming HFSC does a good job interleaving packets in a fine-grained fashion, all you really need to do is focus on "do I have enough bandwidth to handle this traffic". The general rule is below 80% saturation, delay is not an issue. If I want to be able to handle 8Mb/s of VoIP, I really need to just have a bare minimum 10Mb or more bandwidth set aside. Given enough traffic flows, you don't get bursts of data, utilization becomes smooth and predicable. When you don't have enough bandwidth, and bursty traffic comprises a substantial portion of your bandwidth, you'll need to know the "shape" of the traffic you're try work with.

    Individual flows or groups of flows have shapes when there is no congestion. Those are the shapes you want HFSC to mimic when controlling delay. The other elephant in the room is controlling your delay via bursts will always come at the expense of other queues. If VoIP is so important, why not just give that queue enough bandwidth in the first place? If you can't give it enough bandwidth, then no matter what you do, you will get delays because you can't shove 10Mb data down a 5Mb pipe without something giving.

    Bursts seem more useful in the sense of someone has a highly specialized situation with a very specific traffic pattern or working with incredibly slow bandwidth rates where the packet size is large relative to the bandwidth.



  • @Harvy66:

    If VoIP is so important, why not just give that queue enough bandwidth in the first place? If you can't give it enough bandwidth, then no matter what you do, you will get delays because you can't shove 10Mb data down a 5Mb pipe without something giving.

    Bursts seem more useful in the sense of someone has a highly specialized situation with a very specific traffic pattern or working with incredibly slow bandwidth rates where the packet size is large relative to the bandwidth.

    HFSC does not "burst" in the way you think it does. Browse through the HFSC paper. It is only 16 pages, iirc.

    VOIP is one of the major reasons why HFSC is useful. Using the example from the HFSC paper, you can guarantee a  64Kb audio session (160Byte packet sent every 20ms) a delay of 5ms (160Bytes @ 256Kb) instead of 20ms (160Bytes @ 64Kb). I showed the u-max/d-max and m1/d/m2 notation of this exact setup in one of my previous posts.

    You give the VOIP session the delay of a 256Kb (5ms) connection while only allocating an average bitrate of 64Kb, leaving more bandwidth available for applications that are not delay sensitive. VOIP is common today, I would not call it a "highly specialized situation".

    Anyway, I doubt an HFSC misconfiguration could cause your problem, but removing all m1/d entries and using only m2 might be worth trying.

    Hell if I know… :)


  • LAYER 8 Netgate

    Every traffic shaping thread turns into an HFSC theory thread lately.  Can't we just have an HFSC theory thread?



  • @Derelict:

    Every traffic shaping thread turns into an HFSC theory thread lately.  Can't we just have an HFSC theory thread?

    Still seems on topic to me… well, kinda. Traffic-shaper misconfigurations have killed my LAN before.

    I think I have only barged in on 2 or 3 threads with my HFSC evangelism. I am just tired of the spread of HFSC misinformation. I will try to lessen my HFSC posts.

    Edit: If you want to continue the HFSC-centric conversation, head over to https://forum.pfsense.org/index.php?topic=89367.0



  • VOIP is common today, I would not call it a "highly specialized situation".

    If you have a 10Mb+ link that is under 80% utilization, I can guarantee any single 200byte VoIP packet will not be delayed more than ~1ms. I don't need HFSC for delay guarantees if I have enough bandwidth.

    The 256Kb example was an extreme example to show a point. Few people who can afford a professional to come in and analyze their network traffic to the point of understanding how to properly configure HFSC bursts to make best use of their bandwidth for their exact current traffic usage, is not going to be stuck on a 256Kb link.

    Given enough flows, bandwidth usage is predictable, there are no bursts. I've seen statics on 400Gb links that showed them near 95% utilization, yet max queue latency on the link was 0.0ms for an entire week. I'm sure rounded down. For everyone else without 400Gb trunk links and many fewer flows, the 80/20 rule will need to apply.

    Statistically, this is how most network utilization goes. The exact numbers in queue may not apply, but the curve and general idea does.

    Using bursts to exactly control delay will be limited to specialized situations, like low link speeds or non-normal traffic patterns. It's much simpler to think of burst as modifying bandwidth and not caring about how it exactly affects delay with the primary concern of just managing the bandwidth. Even then, burst only really matters once you've consumed all of your normal bandwidth. The good news is that latency sensitive flows tend to also be very low bandwidth.

    HFSC is a powerful tool that can let you get every last drop of bandwidth from your connection while keeping delay low for critical traffic, but I'm perfectly happy with my simplistic view and primarily using it just as a traffic shaper while maintaining 0.1ms pings. All I know is before I had HFSC, my pings were around 20-30ms during link saturation, and after HFSC, about 0.1ms with no special settings. All I did was create a separate 500Kb queue for ICMP and like magic, the ping was kept low. I don't feel I need to understand HFSC any better, at least for what I am capable of testing. My limits of knowledge on the topic may keep me from fulling utilizing it, but with no way of testing, all I an do is guess. Even if I could do better than 0.1ms, I'm not sure it would matter nor would it be useful for every case. Many times specializing for one particular case makes you worse in many other cases. Trying to configure HFSC exactly to maximize one thing could be made useless because of shifting traffic patterns.

    I do enjoy the discussions. I have found several errors in my understanding.



  • The 256Kb/64Kb audio session example from the HFSC paper is actually on a 45Mbit link.

    Whether or not somebody needs HFSC's exclusive features (you probably don't IMO) I cannot know, but if someone is going to use them, use them right, damnit. :)

    Is your LAN still crashing?



  • The detailed level of how exactly HFSC from a math perspective is quite a bit over the head of almost anyone. I would rather take a more statistical view, which is "good enough". You talk about using bursts to control packet ordering because the examples they give lead you down that path with the papers. But who has a 45Mb link and only gives 256Kb/s to VoIP?! Really, that's crazy.

    The level of detail being discussed for their examples does not scale. They are using examples in time resolutions of 10s of milliseconds, when in the real world, you're going to be working in micro and nano seconds. This is magnitudes beyond what people can handle in their mental models. It's like trying to describe how much power is released in a super nova. You can try to use analogies of tons of TNT, but it still does not do it justice. Kilobits per second, yeah, that's fine, but gigabits? Nope.

    When you're working with such short time scales, being accurate can be detrimental to the overall picture. Statistics becomes a better way of handling large numbers of stuff. If you have a low link speed where packets take more than 10ms, you may need to know more about HFSC.



  • @Harvy66:

    The detailed level of how exactly HFSC from a math perspective is quite a bit over the head of almost anyone. I would rather take a more statistical view, which is "good enough". You talk about using bursts to control packet ordering because the examples they give lead you down that path with the papers. But who has a 45Mb link and only gives 256Kb/s to VoIP?! Really, that's crazy.

    The level of detail being discussed for their examples does not scale. They are using examples in time resolutions of 10s of milliseconds, when in the real world, you're going to be working in micro and nano seconds. This is magnitudes beyond what people can handle in their mental models. It's like trying to describe how much power is released in a super nova. You can try to use analogies of tons of TNT, but it still does not do it justice. Kilobits per second, yeah, that's fine, but gigabits? Nope.

    When you're working with such short time scales, being accurate can be detrimental to the overall picture. Statistics becomes a better way of handling large numbers of stuff. If you have a low link speed where packets take more than 10ms, you may need to know more about HFSC.

    You are disagreeing with leaders in the field of networking calculus and statistics.

    I would not be so bold with nothing but anecdotes to support your assertions…

    Just read the paper. Many of your questions and concerns are answered in plain english, some are even answered in the first damn line of the paper... :)
    I read through the paper every time I make an HFSC-related post, to re-confirm my understanding and assure my post is as factual as possible. This is becoming annoying... I would rather we both try to understand HFSC, than you try to guess and I correct you.

    You might find this simple, 2-page paper useful (I did); http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf

    Like Derelict said, let's take our HFSC-centric posts to my "HFSC explained" thread, so we can keep any useful HFSC info in a central place. :)



  • @Nullity:

    @Harvy66:

    The detailed level of how exactly HFSC from a math perspective is quite a bit over the head of almost anyone. I would rather take a more statistical view, which is "good enough". You talk about using bursts to control packet ordering because the examples they give lead you down that path with the papers. But who has a 45Mb link and only gives 256Kb/s to VoIP?! Really, that's crazy.

    The level of detail being discussed for their examples does not scale. They are using examples in time resolutions of 10s of milliseconds, when in the real world, you're going to be working in micro and nano seconds. This is magnitudes beyond what people can handle in their mental models. It's like trying to describe how much power is released in a super nova. You can try to use analogies of tons of TNT, but it still does not do it justice. Kilobits per second, yeah, that's fine, but gigabits? Nope.

    When you're working with such short time scales, being accurate can be detrimental to the overall picture. Statistics becomes a better way of handling large numbers of stuff. If you have a low link speed where packets take more than 10ms, you may need to know more about HFSC.

    You are disagreeing with leaders in the field of networking calculus and statistics.

    I would not be so bold with nothing but anecdotes to support your assertions…

    Just read the paper. Many of your questions and concerns are answered in plain english, some are even answered in the first damn line of the paper... :)
    I read through the paper every time I make an HFSC-related post, to re-confirm my understanding and assure my post is as factual as possible. This is becoming annoying... I would rather we both try to understand HFSC, than you try to guess and I correct you.

    You might find this simple, 2-page paper useful (I did); http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf

    Like Derelict said, let's take our HFSC-centric posts to my "HFSC explained" thread, so we can keep any useful HFSC info in a central place. :)

    I'm not disagreeing with them, I'm just saying math proofs don't tell you how to use HFSC practically, only that it works "as expected" assuming you understand it. 99% of well experienced and educated people in networking will not understand how network traffic works in sub 1ms time scales, so don't even think about time scales that small. I do agree that my "evidence" is anecdotal, I have limited tools. Another term for "anecdotes" is "data points",  albeit with high uncertainty, there is some truth to anecdotes.

    In order to property configure HFSC the way you you've been giving examples, you would need to know these and how these interact:

    1. Packet Size
    2. PPS(Packets per Second) during transmission
    3. Average Bandwidth
    4. Number of flows
    5. Distribution of packets within a flow
    6. Distribution of packets among all flows

    The examples the HFSC links use are simple to prove a point, but do not reflect the complications of a real network. To take the example at face value is to over-simplify the issue, in other words, don't think about individual packets unless that's exactly what happens at the bandwidths you're working with.

    Up to this point I've been arguing an almost laissez faire sort of configuration while you've been arguing an extremely precise. I know I was mostly playing devil's advocate to give an alternative view, but I think the middle ground is best. In this paper(http://www.cs.cmu.edu/~hzhang/papers/SIGCOM97.pdf) that you linked at one point in another thread, it talks about 160byte VoIP packets with an average of 64Kb/s. The way they configured and talked about the burst duration is as a target latency. They set the duration to 5ms as they wanted a 5ms target. m1 was set to the packet size, 160 bytes(not bandwidth like you've mentioned), then spread over the 5ms, they gave the example of 256Kb/s. Because 160bytes/5ms=256Kb.

    What I was able to gain from this example is you set the m1 bandwidth equal to the size you wish to "burst", and the duration to the time in which you wish the extra bytes to be transferred.

    I feel fairly confident that the d(duration), for the purpose of latency sensitive traffic, should be set to your target worst latency. m1 should be thought of not as bandwidth, but the size in bits of the total number of packets relieved during that duration. m2 would be set as the average amount of bandwidth consumed.



  • @Harvy66:

    I'm not disagreeing with them, I'm just saying math proofs don't tell you how to use HFSC practically, only that it works "as expected" assuming you understand it. 99% of well experienced and educated people in networking will not understand how network traffic works in sub 1ms time scales, so don't even think about time scales that small. I do agree that my "evidence" is anecdotal, I have limited tools. Another term for "anecdotes" is "data points",  albeit with high uncertainty, there is some truth to anecdotes.

    In order to property configure HFSC the way you you've been giving examples, you would need to know these and how these interact:

    1. Packet Size
    2. PPS(Packets per Second) during transmission
    3. Average Bandwidth
    4. Number of flows
    5. Distribution of packets within a flow
    6. Distribution of packets among all flows

    The examples the HFSC links use are simple to prove a point, but do not reflect the complications of a real network. To take the example at face value is to over-simplify the issue, in other words, don't think about individual packets unless that's exactly what happens at the bandwidths you're working with.

    **Up to this point I've been arguing an almost laissez faire sort of configuration while you've been arguing an extremely precise. I know I was mostly playing devil's advocate to give an alternative view, but I think the middle ground is best. In this paper(http://www.cs.cmu.edu/~hzhang/papers/SIGCOM97.pdf) that you linked at one point in another thread, it talks about 160byte VoIP packets with an average of 64Kb/s. The way they configured and talked about the burst duration is as a target latency. They set the duration to 5ms as they wanted a 5ms target. m1 was set to the packet size, 160 bytes(not bandwidth like you've mentioned), then spread over the 5ms, they gave the example of 256Kb/s. Because 160bytes/5ms=256Kb.

    What I was able to gain from this example is you set the m1 bandwidth equal to the size you wish to "burst", and the duration to the time in which you wish the extra bytes to be transferred.

    I feel fairly confident that the d(duration), for the purpose of latency sensitive traffic, should be set to your target worst latency. m1 should be thought of not as bandwidth, but the size in bits of the total number of packets relieved during that duration. m2 would be set as the average amount of bandwidth consumed.**

    Incorrect. m1 and m2 define bandwidth.

    @Nullity:

    This quote from the HFSC paper is useful because it clarifies differences between the parameters used in the paper (u-max, d-max, r), and the parameters that pfSense uses (m1, d, m2).

    Each session i is characterized by three parameters: the largest unit of work, denoted u-max, for which the session requires delay guarantee, the guaranteed delay d-max, and the session's average rate r. As an example, if a session requires per packet delay guarantee, then u-max represents the maximum size of a packet. Similarly, a video or an audio session can require per frame delay guarantee, by setting u-max to the maximum size of a frame. The session's requirements are mapped onto a two-piece linear service curve, which for computation efficiency is defined by the following three parameters: the slope of the first segment m1, the slope of the second segment m2, and the x-coordinate of the intersection between the two segments x. The mapping (u-max, d-max, r) -> (m1, x, m2) for both concave and convex curves is illustrated in Figure 8.

    The paper uses u-max (packet/frame size) in it's examples, but pfSense uses m1 (bandwidth).
    When glancing at a configuration using m1, d, and m2, it is obvious whether the parameters are meant to decrease delay (m1 > m2) or increase delay (m1 < m2).
    The u-max/d-max notation is not as intuitive.

    I pulled the quote from the HFSC thread, if you would like to move this HFSC-related conversation there.
    https://forum.pfsense.org/index.php?topic=89367.0

    Edit: Fixed typo in quote from HFSC paper.


Log in to reply