Intelligent shaper



  • Hello
    I need help in building the rules shaping.
    Objective:
    share a channel between 10 users.
    Channel 2 Mbps, getting through the ADSL modem, the modem is configured bridge, and PFSens created 2 PPPoE connections (internet and local network from the ISP).
    I would like to do this:

    1. Priority of HTTP traffic up to 100%, the channel is divided equally between the users dynamically.
      When a user then it up to 100% of the channel, when 2 to 50 and so on …
    2. Priority for other traffic to 80% of the channel. as dynamically shared among users.
    3. (if possible) to separate the torrent traffic is a special rule, and the lowest priority and the channel up to 70%.

    How this should work:
    One (or more), the user downloads a movie (or any of the torrent file) then when any of the users will need to look at the web page of the first rule triggered, and HTTP traffic will select as needed up to 100% of the channel as quickly as possible to download Web pages.

    If the page need to open 2,3 or 5 users then the first rule should be divided channel 2,3 or 5 parts with equal highest priority.

    You. may be surprised, but now this scheme works on ... ADSL modem ACORP 422!
    BUT!
    While the network 5 users and 3 users torrent. and the large number of users this SOHO router is no longer enough.

    Sorry for my English. I'm not very good they possess. I am from Russia:)



  • You can use HFSC to do that by setting the Upperlimit setting in the queues.
    Unfortunately, even L7 cannot catch encrypted torrent traffic.

    For the purpose of the first 2 objectives, you would use HFSC's upperlimit setting.

    So you'll set 2 rules to channel HTTP and HTTPS traffic to qOthersHigh.
    Under qDefault (which captures everything else), you'll set Upperlimit M2 to 80%.

    I am not sure if the L7 (pfsense 2.0) is working for traffic shaping at the moment but if it actually works, then you'll set it to pipe torrent traffic to qOthersLow and set the upperlimit for this queue to 70%.



  • Could you give an example of the necessary filters?
    Do not consider me lazy, I just work with PFSens first time, and with fribsd also the first time.

    And the main problem: it is difficult to study the technical literature in English.

    I would be very grateful to have responded!



  • Which version of pfsense are you using?  The shaper configuration is very different between 1.2.x and 2.0



  • Sorry, I forgot to specify!
    Version of pfSense-2.0-BETA4-20100821-2136.

    As far as I know, version 1.2 … problem with shaping traffic connecting through PPPoe.
    So I chose the version 2.0 ....



  • @Tzar:

    How this should work:
    One (or more), the user downloads a movie (or any of the torrent file) then when any of the users will need to look at the web page of the first rule triggered, and HTTP traffic will select as needed up to 100% of the channel as quickly as possible to download Web pages.

    With HFSC, this is impossible. The movie / torrent downloader always get its bandwidth share. So the web browser can never get 100% bandwidth when the downloader is still active.



  • Hrmm..  I do suppose he can allocate say 0Kb 200 1% bandwidth share for the bulk/ default queue at a lower priority and carefully adjusting the bandwidth share curve for the rules with higher priority.  That should give the download queue no share for the first 200ms, which should be enough for a quick burst on web traffic until the line comes close to saturation.



  • Priority to 100% is not mandatory, it is possible 80-90%.

    I only ask for help in constructing the initial configuration in the future, I perfected it:)

    At this point in my modem, this is done using the algorithm of the srr.



  • @dreamslacker:

    Hrmm..  I do suppose he can allocate say 0Kb 200 1% bandwidth share for the bulk/ default queue at a lower priority and carefully adjusting the bandwidth share curve for the rules with higher priority.  That should give the download queue no share for the first 200ms, which should be enough for a quick burst on web traffic until the line comes close to saturation.

    The download queue then allocated 1% share, there remains only 99% for others.



  • @Tzar:

    Priority to 100% is not mandatory, it is possible 80-90%.

    I only ask for help in constructing the initial configuration in the future, I perfected it:)

    At this point in my modem, this is done using the algorithm of the srr.

    Here is my config. It works under 2.0. Pls don't try it under 1.2.x.

    This is a neutral policy (prefer nothing, disprefer nothing), suitable for ISPs / enterprises. You can try to add some upper limit to qP2P if you like.

    Priority Queue           B/W     Link-share      Real-time    Upper-limit
                                  m1    d   m2   m1    d   m2   m1    d   m2
    ======== ============== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
          1 qP2P             5%                            1%
          2 qOthersLow      10%                            1%
          3 qOthersDefault  20%                            1%
          4 qOthersHigh     20%                            8%
          5 qGames          20%                            4%
          6 qACK             5%                            2%
          7 qVoIP           20%                            4%
    ======== ============== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
            TOTAL          100%                           21%

    Notes:

    – qP2P is the default queue. Bittorrent and the likes go there.

    -- qOthersLow: mail, news, chat and some bulk download protocols.

    -- qOthersDefault: HTTP, HTTPS (except firewall control traffics).

    -- qOthersHigh: ICMP, DNS, SSH, HTTP/HTTPS firewall control traffics, VPNs.

    -- qGames: games and NTP.

    -- qVoIP: interactive audio and video.

    Edit: oops I overlooked that you are using 2mbps ADSL which is likely 2mbps downlink and 512 - 640 kbps uplink. Pls refer to the sticky topic on ACK queue sizing for asymmetric link on how to adjust qACK. 5% does not suffice.



  • @dusan:

    The download queue then allocated 1% share, there remains only 99% for others.

    If the line is saturated and the other queues do use up the remaining bandwidth, yes.  However, HFSC has borrowing capability and you can set the other queues to have less allocated bandwidth share.
    i.e.  When higher priority traffic is cleared (web traffic is generally bursty and should mostly clear within the 'd' timelimit set; some tweaking required here), the download queue can still borrow the excess bandwidth up to the upperlimit.



  • @dreamslacker:

    @dusan:

    The download queue then allocated 1% share, there remains only 99% for others.

    If the line is saturated and the other queues do use up the remaining bandwidth, yes.  However, HFSC has borrowing capability and you can set the other queues to have less allocated bandwidth share.
    i.e.  When higher priority traffic is cleared (web traffic is generally bursty and should mostly clear within the 'd' timelimit set; some tweaking required here), the download queue can still borrow the excess bandwidth up to the upperlimit.

    The torrent downloader can borrow bandwidth from the web browser and this happens to be the typical case, yes.

    The OP's concern is, however, whether the web browser can borrow bandwidth from the torrent downloader. It can't. That's because downloading is, as opposed to browsing, continually backlogged (i.e. it is active all the time).

    And this is true (almost) regardless of queue's priority since priority is (again almost) ignored in HFSC decision making.



  • @dreamslacker:

    Unfortunately, even L7 cannot catch encrypted torrent traffic.

    Level 7 filter can't classify everything even if it's unencrypted, since BitTorrent detection is not perfect. Whitelisting is the possible solution, but it will probably interfere with Skype detection - Skype also doesn't use fixed ports and can be hard to detect too.



  • Then perhaps CBQ would be a better choice for him.  With the Web traffic queue taking a 20% bandwidth and higher priority with borrow.  That leaves 80% for borrowing on other queues when the high prio. queue doesn't need the b/w.

    Just to update:

    I reckon, he could classify HTTP/ HTTPS/ SMTP/ POP3 type traffic into qOthersHigh and leave the Default queue as the catchall to catch torrent traffic (Whitelist system as suggested by bortmex).

    For algorithm, CBQ/ PRIQ might work better for him albeit without transfer rate limiting (Downstream shaping with HFSC is a bit of a black art to begin with).

    Hence, for CBQ, we will see something like this:

    WAN

    • qACK - 5% Priority 7 Borrow

    • qOthersHigh - 10% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    LAN

    • qACK - 4Kb Priority 7 Borrow

    • qOthersHigh - 15% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    Needs a fair bit of tweaking after that especially if the line is highly asymmetric.  I'm assuming a 2Mb/ 512Kb line here.

    Last but not least, he actually seems to require simple strict priority unfair sharing QoS.  i.e. PRIQ. which is what is provided on most SoC based consumer routers as Hi, Mid, Lo.



  • @dusan:

    The torrent downloader can borrow bandwidth from the web browser and this happens to be the typical case, yes.

    The OP's concern is, however, whether the web browser can borrow bandwidth from the torrent downloader. It can't. That's because downloading is, as opposed to browsing, continually backlogged (i.e. it is active all the time).

    And this is true (almost) regardless of queue's priority since priority is (again almost) ignored in HFSC decision making.

    On a side note, I actually got this to work on 1.2.2 & 1.2.3 which is why I suggested this.  However, shaping the downstream is somewhat of a blackart.
    Rather than use the Linkshare on the default queue, I calculated and set the Linkshare only for the other queues (qACK, qGames) instead.
    Thus, the default queue doesn't have the ability to override the priority (or so it seems since it worked exactly that way for me).



  • @dreamslacker:

    Then perhaps CBQ would be a better choice for him.  With the Web traffic queue taking a 20% bandwidth and higher priority with borrow.  That leaves 80% for borrowing on other queues when the high prio. queue doesn't need the b/w.

    Just to update:

    I reckon, he could classify HTTP/ HTTPS/ SMTP/ POP3 type traffic into qOthersHigh and leave the Default queue as the catchall to catch torrent traffic (Whitelist system as suggested by bortmex).

    For algorithm, CBQ/ PRIQ might work better for him albeit without transfer rate limiting (Downstream shaping with HFSC is a bit of a black art to begin with).

    Hence, for CBQ, we will see something like this:

    WAN

    • qACK - 5% Priority 7 Borrow

    • qOthersHigh - 10% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    LAN

    • qACK - 4Kb Priority 7 Borrow

    • qOthersHigh - 15% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    Needs a fair bit of tweaking after that especially if the line is highly asymmetric.  I'm assuming a 2Mb/ 512Kb line here.

    Last but not least, he actually seems to require simple strict priority unfair sharing QoS.  i.e. PRIQ. which is what is provided on most SoC based consumer routers as Hi, Mid, Lo.

    I finally started to build a router for your advice on the CBQ algorithm , and immediately the question arose how to divide the channel for ADSL 2000/512kbps?

    Sorry, but I could not find it in search!



  • @Tzar:

    I finally started to build a router for your advice on the CBQ algorithm , and immediately the question arose how to divide the channel for ADSL 2000/512kbps?

    Sorry, but I could not find it in search!

    You don't need to.  TCP/IP already handles that (sharing).  You could, of course, limit how much a certain IP can bombard the connection by setting the CPS limits in the firewall rules (set one individual rule for each client IP and restrict the max CPS; repeat for all the rules).



  • The torrent downloader can borrow bandwidth from the web browser and this happens to be the typical case, yes.

    The OP's concern is, however, whether the web browser can borrow bandwidth from the torrent downloader. It can't. That's because downloading is, as opposed to browsing, continually backlogged (i.e. it is active all the time).

    And this is true (almost) regardless of queue's priority since priority is (again almost) ignored in HFSC decision making  http://thuocdongduoc.vn/

    On a side note, I actually got this to work on 1.2.2 & 1.2.3 which is why I suggested this.  However, shaping the downstream is somewhat of a blackart.
    Rather than use the Linkshare on the default queue, I calculated and set the Linkshare only for the other queues (qACK, qGames) instead.
    Thus, the default queue doesn't have the ability to override the priority (or so it seems since it worked exactly that way for me



  • @dreamslacker:

    Then perhaps CBQ would be a better choice for him.  With the Web traffic queue taking a 20% bandwidth and higher priority with borrow.  That leaves 80% for borrowing on other queues when the high prio. queue doesn't need the b/w.

    Just to update:

    I reckon, he could classify HTTP/ HTTPS/ SMTP/ POP3 type traffic into qOthersHigh and leave the Default queue as the catchall to catch torrent traffic (Whitelist system as suggested by bortmex).

    For algorithm, CBQ/ PRIQ might work better for him albeit without transfer rate limiting (Downstream shaping with HFSC is a bit of a black art to begin with).

    Hence, for CBQ, we will see something like this:

    WAN

    • qACK - 5% Priority 7 Borrow

    • qOthersHigh - 10% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    LAN

    • qACK - 4Kb Priority 7 Borrow

    • qOthersHigh - 15% Priority 4 Borrow ECN

    • qDefault - 1Kb Priority 2 Borrow ECN RED

    Needs a fair bit of tweaking after that especially if the line is highly asymmetric.  I'm assuming a 2Mb/ 512Kb line here.

    Last but not least, he actually seems to require simple strict priority unfair sharing QoS.  i.e. PRIQ. which is what is provided on most SoC based consumer routers as Hi, Mid, Lo.

    I'm trying to do basically the same thing as the OP on a pf 1.2.3 setup.  I understand everything here except the "Borrow" piece.  Is that the "link share" setting in 1.2.3 or is it a purely 2.0 feature?



  • @josefry:  The borrow setting is part of CBQ, not HFSC.  HFSC already has borrow capabilities by default.

    In CBQ, you set a certain amount of bandwidth guarantee a queue.  This is the bandwidth setting.  If you do not allow borrowing, then this queue is limited to the bandwidth you set.  If borrowing is allowed, then the queues can borrow available (spare) bandwidth from the parent in order of priority.
    i.e.  if 2 queues need to borrow, the one with the higher priority gets to borrow what it needs first then the remaining (if any) is given to the next queue.



  • Thank you!  That makes complete sense to me now.


Locked