Traffic shaper changes [90% completed, please send money to complete bounty]



  • This is a continuation of http://forum.pfsense.org/index.php/topic,2686.0.html

    The following are proposed changes to the existing pfSense traffic shaper based on community interest, feasibility, and pledged dollars.  The list is somewhat out of order of proposals due to references made in the list to other items.

    • Multi-interface: Allow the shaper to work with multiple LAN (LAN, OPT1, OPTx) interfaces and a single WAN connection.

    • Bridges:  Make the shaper code work with bridging - I'm sure I can do this for two interfaces, I'm not sure I can make this work in combination with multi-interface shaping or with more than two bridge interfaces.

    • Multi-WAN shaping: under consideration - I need to give this more thought, I'm not sure if it's feasible w/in the scope of our UI (or with how we have to get traffic into queues).

    • Wizard updates to make the above changes easy to configure.

    • UI updates to make the shaper settings easier (easier, not necessarily easy) to modify

    I fully expect that we'll be able to get these changes in the next feature release, however there is no chance that they'll get into anything with a 1.0.x version and I think we're targetting 1.1 for a FreeBSD OS update.  This caveat is just to help set expectations so when we release a 1.0.2 or 1.1 nobody asks whether these changes made it in.

    I'd like to ask that those that pledged in the other thread (that I'll merge into this one) please reply with an update on whether the proposal meets their expectations and an updated pledge which I'll add to the above list.

    There is no "price tag" for this feature enhancement, nor is there a time frame other than what I mentioned above (it'd still have to be debugged and thoroughly tested even if I wrote the code in a day).  The $10k pledge drive that was initiated was to drive interest, not as a price tag.  All I ask is for those that are interested to put their $$$ where their mouths are and help get this going.  I've put up with a lot of crap over the existing code and am kind of tired of people bitching about it; if people want it to work better, I need to see it.  Some people have certainly showed interest, with the amount of people that have downloaded and are using pfSense, I'm hoping others will help the cause here and won't leave this up to the half dozen that have already showed their interest.

    Bounties pledged to date:

    • $50 - pfSense

    • $100 - sai

    • $200 - wcoolnet

    • $500 - mrt_ok (received in paypal account)

    • $100 - Delphinus

    Total pledged: $950

    Remember, pledging is only part of the equation, we don't want to be told we're going to get X amount of dollars and not get it after all the work is done.  I request that at least 50% of each pledge be escrowed in the pfSense account (you'll have to trust that I don't have access to it and that Scott isn't going to go and get really really drunk with it).  Thanks

    Update - 11/22/2006 - I've been working on the pf.conf side of this change a bit and think I've made some progress.  I should begin coding these changes shortly as it's getting quite painful to make any further progress by hand.  As soon as I have something that can be tested, I'll contact those that have made pledges on this bounty to allow them to test.

    –Bill



  • $100.

    Multi-wan is interesting, but what do you mean by that? If you mean 1 shaper with multiple WANs then I dont remember seeing it in any of the HFSC docs (not that I studied too hard). It would probably be very difficult to visualise and use  - most unlikely anyone would want it.
    If you mean a separate shaper for each WAN then that makes more sense. Should be possible to fit it into the gui without complications.

    Time line: are we talking 6 months, 1 year (+/- x months)?  I know that this is open source so no release dates set in advance, but some sort of ETA ?



  • @sai:

    $100.

    Multi-wan is interesting, but what do you mean by that? If you mean 1 shaper with multiple WANs then I dont remember seeing it in any of the HFSC docs (not that I studied too hard). It would probably be very difficult to visualise and use  - most unlikely anyone would want it.
    If you mean a separate shaper for each WAN then that makes more sense. Should be possible to fit it into the gui without complications.

    Yep, multiple WANs.  It's all about queue definition and how traffic get's assigned to the queue.  I'm not 100% sure I can make it work in our case.  Policy based multi-wan routing would likely be possible, but based on how we assign traffic to queues (which won't be changing…it's the only way we can do it without major pf hacks) it's likely that load balanced WANs won't be able to shape.  There are some other items I have in the works as I get to them (kernel side is done...I think) that will allow for layer7 requeueing of traffic (ie. traffic starts off in the default queue or something and get's moved to HTTP once it recognizes HTTP in the packet payload).  I'm not including that as part of this bounty/feature enhancement.

    @sai:

    Time line: are we talking 6 months, 1 year (+/- x months)?  I know that this is open source so no release dates set in advance, but some sort of ETA ?

    We're planning on starting work on RELENG_2 shortly and I'm (personally) hopeful that we'll be able to push a 6 month release cycle out (but depending on bugs in new features, that might stretch).  With that said, you'll be able to see the new code in the alpha releases as soon as it's commited.  I don't expect that it'll take me more than a week or two to get the new shaper code to a point where I can commit it and get people pounding on it.

    –Bill



  • $200



  • $500



  • bill the changes you are proposing sound very interesting. if interfaces could be extended to support all the ng interfaces loaded on the pppoe server we have another 200 for your bounty.

    it can nearly do it now. with a bit of a hack to it but gets it's white spacings wrong.

    would you think this is supportable in the multiple interfaces area of your plan



  • @aldo:

    bill the changes you are proposing sound very interesting. if interfaces could be extended to support all the ng interfaces loaded on the pppoe server we have another 200 for your bounty.

    it can nearly do it now. with a bit of a hack to it but gets it's white spacings wrong.

    would you think this is supportable in the multiple interfaces area of your plan

    I think so - do PPPOE server interfaces show up as individual interfaces in the Rules (or even interfaces) screen?  I'm thinking it doesn't, but I haven't really seen how the PPPoE server works either.  I'll try and spend some time tonight, I agree, it's likely possible, but I don't know enough about how that section of code works to be able to say for sure.  If it comes up as individual interfaces that rules can be created on, then I'm reasonably confident that it'll "just work".

    –Bill



  • Only major problem is ngX is dynamic.  The ordering may shift.



  • @sullrich:

    Only major problem is ngX is dynamic.  The ordering may shift.

    Good point.  Although I don't think ngX necessarily has to be dynamic.  But making major interface changes, while on my own personal list of things to do, aren't necessarily compatible within the scope of this change.  If these are truly dynamic and not tied to the standard rules editor, then I don't think we'll be able to make it part of this change.  Although this change should set the stage for this feature in the future.

    –Bill



  • Nope, its not associated unfortunately.



  • ok that was confusing could you clarify these points.
    is it possible?
    would you include it?
    all i am trying to do is hfsc with them no other gaurentees all have equal preferance.

    well look forward to the clarification



  • @aldo:

    ok that was confusing could you clarify these points.
    is it possible?

    Possible, yes.  With the current way interfaces are configured, no.  The shaper changes I'm working on won't directly help here, but would be considered a prereq to being able to do this.

    @aldo:

    would you include it?

    One thing at a time :)  If PPPoE server 'nics' (all the ng interfaces) were already individually assignable for rule management in pfSense, you'd get the shaper changes "for free" so to speak.  The changes I'm looking at would just come along for the ride.  As it sits, I'd consider this a different project, but one that relies on this change before it can be seriously thought of.  Depending on how the code ends up getting written, it may be possible to hack up a config.xml that'll create the correct rules - not sure, I'm still researching the proper way to do the queues as it is (it's looking like we'll have a number of nasty recursive loops).

    @aldo:

    all i am trying to do is hfsc with them no other gaurentees all have equal preferance.

    well look forward to the clarification

    Hope that helps.

    –Bill



  • Not sure that we can accept Pakistan funds without Big Bubba getting down and angry with us.



  • i too would be willing to throw some money your way for this, however seeing as I'm pretty close to broke it wouldn't be much.



  • I'll donate $100. Let me know when you need it.



  • Just sent in $100 to Paypal…while I would like you to consider it part of this bounty, please use it as/when the project needs...you've earned it with or without a multi-interface traffic shaper!



  • Bill what paypal account do i send my donation to?



  • @billm:

    @aldo:

    ok that was confusing could you clarify these points.
    is it possible?

    Possible, yes.  With the current way interfaces are configured, no.  The shaper changes I'm working on won't directly help here, but would be considered a prereq to being able to do this.

    @aldo:

    would you include it?

    OK i think i understand what are the overall thoughts on this. should i start up a bounty on it.
    we use pppoe server for all our wireless concentration. if this change looks achievable outside of the shaper scope i will make a bounty for it.

    maybe you or scott can clarify the scope of the change a little more clearly and i can brief it

    One thing at a time :)  If PPPoE server 'nics' (all the ng interfaces) were already individually assignable for rule management in pfSense, you'd get the shaper changes "for free" so to speak.  The changes I'm looking at would just come along for the ride.  As it sits, I'd consider this a different project, but one that relies on this change before it can be seriously thought of.  Depending on how the code ends up getting written, it may be possible to hack up a config.xml that'll create the correct rules - not sure, I'm still researching the proper way to do the queues as it is (it's looking like we'll have a number of nasty recursive loops).

    @aldo:

    all i am trying to do is hfsc with them no other gaurentees all have equal preferance.

    well look forward to the clarification

    Hope that helps.

    –Bill



  • @Perry:

    Bill what paypal account do i send my donation to?

    paypal at chrisbuechler.com if you want pfSense to hold onto it until I'm done, or billm at pfsense.org if you wish to send it direct to me sooner.

    –Bill



  • Aldo, didn't see any content in that post…did I miss something?

    --Bill

    @aldo:

    @billm:

    @aldo:

    ok that was confusing could you clarify these points.
    is it possible?

    Possible, yes.  With the current way interfaces are configured, no.  The shaper changes I'm working on won't directly help here, but would be considered a prereq to being able to do this.

    @aldo:

    would you include it?

    OK i think i understand what are the overall thoughts on this. should i start up a bounty on it.
    we use pppoe server for all our wireless concentration. if this change looks achievable outside of the shaper scope i will make a bounty for it.

    maybe you or scott can clarify the scope of the change a little more clearly and i can brief it

    One thing at a time :)  If PPPoE server 'nics' (all the ng interfaces) were already individually assignable for rule management in pfSense, you'd get the shaper changes "for free" so to speak.  The changes I'm looking at would just come along for the ride.  As it sits, I'd consider this a different project, but one that relies on this change before it can be seriously thought of.  Depending on how the code ends up getting written, it may be possible to hack up a config.xml that'll create the correct rules - not sure, I'm still researching the proper way to do the queues as it is (it's looking like we'll have a number of nasty recursive loops).

    @aldo:

    all i am trying to do is hfsc with them no other gaurentees all have equal preferance.

    well look forward to the clarification

    Hope that helps.

    –Bill



  • Just wanted to update the thread.  I'm still working on this, had some issues with some of the new gui libraries that we needed to get fixed as well as some VM issues that are now resolved.  I'm hoping to spend some time during my vacation to get a new wizard completed which should allow me to generate configs that I can use to create the backend :)  Due to the use of the new gui library, I can pretty easily say that this won't appear in the RELENG_1 branch at all, but I'll attempt to backport it for those that have pledged and donated for this so it can get tested and have some eyes on it earlier (and of course so you can have a new toy :)).

    –Bill



  • Thanks Bill.

    I have a request: could you make the wizard optional please?

    I realize that altq is really difficult to understand, but sometimes you just want to set things up yourself. This is especially true when you are trying to learn about the software.



  • @sai:

    Thanks Bill.

    I have a request: could you make the wizard optional please?

    I realize that altq is really difficult to understand, but sometimes you just want to set things up yourself. This is especially true when you are trying to learn about the software.

    The wizard is already optional.  I do plan on making the manual configuration a little more reliable and less prone to easy breakage (the real problem) though.

    –Bill



  • Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.



  • @Leoandru:

    Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.

    You might check out http://wiki.pfsense.com/wikka.php?wakka=NewShaperNotes.  I think I can handle bridge, and multi-lan w/out too much problem.  Multi-wan is going to be a tad more challenging I think.

    –Bill



  • @billm:

    @Leoandru:

    Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.

    You might check out http://wiki.pfsense.com/wikka.php?wakka=NewShaperNotes.  I think I can handle bridge, and multi-lan w/out too much problem.  Multi-wan is going to be a tad more challenging I think.

    –Bill

    cool, I'll experiment with altq and multi-wan shaping and update the wiki with my findings and ideas. Off the bat though I'm not sure if this can be done without modifying altq itself. Also I'll experiment with the ideas you currently have to see if I can add any additional info. What about transparent/l7 shaping? have any ideas or wiki entry on that? I have a few idea's I'd like to share on that, I probably make a wiki entry once I setup a testing platform this weekend and put together some notes.



  • @Leoandru:

    cool, I'll experiment with altq and multi-wan shaping and update the wiki with my findings and ideas. Off the bat though I'm not sure if this can be done without modifying altq itself. Also I'll experiment with the ideas you currently have to see if I can add any additional info. What about transparent/l7 shaping? have any ideas or wiki entry on that? I have a few idea's I'd like to share on that, I probably make a wiki entry once I setup a testing platform this weekend and put together some notes.

    Yay!  Glad to see you have some free time Leo!



  • just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.



  • Thanks Leon…I'll check out the wiki, the configs apply, but I'm not terribly surprised it doesn't work quite as advertised :-/

    --Bill



  • As dummynet can shape incoming on an interface this would be an option to shape traffic inside tunnels as well (before the traffic on the outgoing interface is only seen as encrypted traffic only). I have some setups that work this way pretty well with m0n0wall. However, getting this all under control and even crunching all that logic in a wizard will be a hard task I guess and considering multiple interfaces…



  • @hoba:

    As dummynet can shape incoming on an interface this would be an option to shape traffic inside tunnels as well (before the traffic on the outgoing interface is only seen as encrypted traffic only). I have some setups that work this way pretty well with m0n0wall. However, getting this all under control and even crunching all that logic in a wizard will be a hard task I guess and considering multiple interfaces…

    Dummynet does not work with ALTQ/PF.  As soon as you add a RDR, all traffic stops on the firewall.



  • @Leoandru:

    just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.

    Leon, any updates on this?  I've been holding off spending much more time on this until it's proven working (or not)…it should work I think, but it's a bit of a hack to setup as best as I can tell.

    --Bill



  • @billm:

    Leon, any updates on this?  I've been holding off spending much more time on this until it's proven working (or not)…it should work I think, but it's a bit of a hack to setup as best as I can tell.

    --Bill

    No.. I haven't been able to make it work, I was holding off the write up on this until I had absolutely given up. Also I didn't know your were holding off until more proof could be given that it doesn't work, but I was hoping that you could prove me wrong with some test and sample setups. I was still experimenting with several ideas  though I haven't gotten as much time as I would have liked to experiment with them (maybe i spoke too soon of free time cause it seems to be vanishing into work). Maybe this weekend I'll be able to give something more concrete, but please go ahead with your ideas and experiment I check this thread regularly for updates so you can post any success you have had with this. Sorry for the lack of correspondence on irc, it would be nice if we could bounce ideas off each other but I just havn't found the time yet.



  • 'k I'll just drop you a private email if I can find your address again :)

    –Bill



  • I would contribute $25 for proper dual-wan QoS/shaping.



  • just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.

    So does this mean that it's not possible to shape across multiple WAN interfaces? Or does it mean that we can't even shape across a bridged WAP and LAN connected to a single WAN..  ???



  • Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.



  • No.



  • @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.



  • @cabe:

    @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.

    Have fun!


Locked