Traffic shaper changes [90% completed, please send money to complete bounty]



  • Aldo, didn't see any content in that post…did I miss something?

    --Bill

    @aldo:

    @billm:

    @aldo:

    ok that was confusing could you clarify these points.
    is it possible?

    Possible, yes.  With the current way interfaces are configured, no.  The shaper changes I'm working on won't directly help here, but would be considered a prereq to being able to do this.

    @aldo:

    would you include it?

    OK i think i understand what are the overall thoughts on this. should i start up a bounty on it.
    we use pppoe server for all our wireless concentration. if this change looks achievable outside of the shaper scope i will make a bounty for it.

    maybe you or scott can clarify the scope of the change a little more clearly and i can brief it

    One thing at a time :)  If PPPoE server 'nics' (all the ng interfaces) were already individually assignable for rule management in pfSense, you'd get the shaper changes "for free" so to speak.  The changes I'm looking at would just come along for the ride.  As it sits, I'd consider this a different project, but one that relies on this change before it can be seriously thought of.  Depending on how the code ends up getting written, it may be possible to hack up a config.xml that'll create the correct rules - not sure, I'm still researching the proper way to do the queues as it is (it's looking like we'll have a number of nasty recursive loops).

    @aldo:

    all i am trying to do is hfsc with them no other gaurentees all have equal preferance.

    well look forward to the clarification

    Hope that helps.

    –Bill



  • Just wanted to update the thread.  I'm still working on this, had some issues with some of the new gui libraries that we needed to get fixed as well as some VM issues that are now resolved.  I'm hoping to spend some time during my vacation to get a new wizard completed which should allow me to generate configs that I can use to create the backend :)  Due to the use of the new gui library, I can pretty easily say that this won't appear in the RELENG_1 branch at all, but I'll attempt to backport it for those that have pledged and donated for this so it can get tested and have some eyes on it earlier (and of course so you can have a new toy :)).

    –Bill



  • Thanks Bill.

    I have a request: could you make the wizard optional please?

    I realize that altq is really difficult to understand, but sometimes you just want to set things up yourself. This is especially true when you are trying to learn about the software.



  • @sai:

    Thanks Bill.

    I have a request: could you make the wizard optional please?

    I realize that altq is really difficult to understand, but sometimes you just want to set things up yourself. This is especially true when you are trying to learn about the software.

    The wizard is already optional.  I do plan on making the manual configuration a little more reliable and less prone to easy breakage (the real problem) though.

    –Bill



  • Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.



  • @Leoandru:

    Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.

    You might check out http://wiki.pfsense.com/wikka.php?wakka=NewShaperNotes.  I think I can handle bridge, and multi-lan w/out too much problem.  Multi-wan is going to be a tad more challenging I think.

    –Bill



  • @billm:

    @Leoandru:

    Hey Bill mind if I chip in on this project? I'm finding more free time on my hand these days, so I'm specifically interested in helping with transparent shaping and investigating the muliwan/multinterface shaping of altq.

    You might check out http://wiki.pfsense.com/wikka.php?wakka=NewShaperNotes.  I think I can handle bridge, and multi-lan w/out too much problem.  Multi-wan is going to be a tad more challenging I think.

    –Bill

    cool, I'll experiment with altq and multi-wan shaping and update the wiki with my findings and ideas. Off the bat though I'm not sure if this can be done without modifying altq itself. Also I'll experiment with the ideas you currently have to see if I can add any additional info. What about transparent/l7 shaping? have any ideas or wiki entry on that? I have a few idea's I'd like to share on that, I probably make a wiki entry once I setup a testing platform this weekend and put together some notes.



  • @Leoandru:

    cool, I'll experiment with altq and multi-wan shaping and update the wiki with my findings and ideas. Off the bat though I'm not sure if this can be done without modifying altq itself. Also I'll experiment with the ideas you currently have to see if I can add any additional info. What about transparent/l7 shaping? have any ideas or wiki entry on that? I have a few idea's I'd like to share on that, I probably make a wiki entry once I setup a testing platform this weekend and put together some notes.

    Yay!  Glad to see you have some free time Leo!



  • just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.



  • Thanks Leon…I'll check out the wiki, the configs apply, but I'm not terribly surprised it doesn't work quite as advertised :-/

    --Bill



  • As dummynet can shape incoming on an interface this would be an option to shape traffic inside tunnels as well (before the traffic on the outgoing interface is only seen as encrypted traffic only). I have some setups that work this way pretty well with m0n0wall. However, getting this all under control and even crunching all that logic in a wizard will be a hard task I guess and considering multiple interfaces…



  • @hoba:

    As dummynet can shape incoming on an interface this would be an option to shape traffic inside tunnels as well (before the traffic on the outgoing interface is only seen as encrypted traffic only). I have some setups that work this way pretty well with m0n0wall. However, getting this all under control and even crunching all that logic in a wizard will be a hard task I guess and considering multiple interfaces…

    Dummynet does not work with ALTQ/PF.  As soon as you add a RDR, all traffic stops on the firewall.



  • @Leoandru:

    just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.

    Leon, any updates on this?  I've been holding off spending much more time on this until it's proven working (or not)…it should work I think, but it's a bit of a hack to setup as best as I can tell.

    --Bill



  • @billm:

    Leon, any updates on this?  I've been holding off spending much more time on this until it's proven working (or not)…it should work I think, but it's a bit of a hack to setup as best as I can tell.

    --Bill

    No.. I haven't been able to make it work, I was holding off the write up on this until I had absolutely given up. Also I didn't know your were holding off until more proof could be given that it doesn't work, but I was hoping that you could prove me wrong with some test and sample setups. I was still experimenting with several ideas  though I haven't gotten as much time as I would have liked to experiment with them (maybe i spoke too soon of free time cause it seems to be vanishing into work). Maybe this weekend I'll be able to give something more concrete, but please go ahead with your ideas and experiment I check this thread regularly for updates so you can post any success you have had with this. Sorry for the lack of correspondence on irc, it would be nice if we could bounce ideas off each other but I just havn't found the time yet.



  • 'k I'll just drop you a private email if I can find your address again :)

    –Bill



  • I would contribute $25 for proper dual-wan QoS/shaping.



  • just a little update: The multiple interface shaping feature is starting to look a bit daunting, altq was not designed for it. The queuing hierarchy created on each interface are totally unrelated. So if you try to shape 1 wan interface over two lans then altq simple can't do it. Probably some combination of dummynet and altq would solve the problem, I'll post my opinions on the wiki later.

    So does this mean that it's not possible to shape across multiple WAN interfaces? Or does it mean that we can't even shape across a bridged WAP and LAN connected to a single WAN..  ???



  • Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.



  • No.



  • @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.



  • @cabe:

    @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.

    Have fun!



  • @cabe:

    @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.

    This is really helpfull, I guess we have to question the whole project now that you are gone  :o



  • I will use this as a catch all qos bountys section.

    I would join in with 200$ for a multi interfaces Qos and multi PPPoE Wan support. They are only useful together for me.
    Additionally, I would pay 100$ for per IP bandwith limitations in the traffic shaper.
    I know all this is not much for the lot of work it will require, but maybe others will join in.

    Greetings,
    techatdd



  • @hoba:

    @cabe:

    @eickst:

    Any updates on this feature?  Not multi-wan but at least multi-lan such as WAP and LAN.

    I'm probably going to just switch to the new beta of m0n0wall for that feature.

    This is really helpfull, I guess we have to question the whole project now that you are gone  :o

    Well I'm glad you think my comment was helpful, because I certainly intended for it to be! You shouldn't question the project though – It's a great project and I enjoy seeing it progress. Oh, and I'm not gone either :)

    eickst was looking for a multi-lan traffic shaper, and I suggested m0n0wall. I know that it supports multi-lan traffic shaping, so it could be of use to him.

    I also mentioned m0n0wall's new beta because it supports Atheros cards, and since I have an Atheros card, I haven't been able to use the latest version of m0n0wall since they switched back to 4.x since the 1.2x versions. I see that the new m0n0wall beta supports Atheros wireless cards, thus with no advancement in pfSense's multi-lan trafic shaper (and no complaints coming from me about that), I'm going to give m0n0wall a shot.

    It's like using the appropriate tool for the job.  Just because you choose to use a screwdriver to screw in a screw doesn't mean you think that hammers suck!



  • adding 100$ for shapper work on a dual wan balenced pool . (with failover of course ! :-)

    rgrds



  • Is this bounty still open?

    I'll pony up $100. An IP/MAC-based shaper would be my top request but any movement deserves a contribution.

    Feel free to punt this late post if I've missed an update elsewhere.



  • Can someone please summerize (maybe you Bill), what the current status is and what the problems are?

    I think it would be easier for all to help….

    thank you!



  • Can someone advise if this bounty is still open?

    I would like to post an additional 50$ bounty for Wulti-Wan traffic shaping, with individual-WAN traffic-shaping settings alongwith a feature that allows one to add an undefined type of traffic by port or originating/destination IP and create a custom queue for it e.g. one can allocate lower priority for a protocol that has not been specifically defined in gui…say rsync or a lower priority for my software repository server updating its various repos continuously.

    With best regards.
    Sanjay.



  • About the multi-wan, I got a question.

    The multi-wan, would have two separate queue for each of the WAN interface, and they would be independent correct?

    Is there any way that it could be possible to link them together?  Let's say I have two WAN, and both goes over the same line, so if one WAN eats up 500kbps, that is 500kbps less that will be allocated to the other WAN, is this possible?



  • Hi,

    As asked before, is this bounty still open?, I need this feature and I'll like to post some money, but I didn't see any comment from the bounty leader for a long time.

    Regards
    Alfredo



  • Could somebody tell me if this bounty is still open

    Appreciate very much your comments.

    Regards
    Alfredo



  • This for sure is still a hot topic though not trivial to solve. Maybe we can discuss it during this years hackathon, which will take place in october, to get some things rolling.



  • Thanks Hoba,

    And how can help with it?, I'm not an expert, but I can help testing and with some money, how can I send it?

    Regards
    Alfredo



  • Is multi-WAN traffic shaping still "under consideration"?

    Also, is that under consideration for the feature of having a single queue for all outbound traffic? I would be happy enough for separate queues for each line and I feel that that should already be pretty easy to do (just change interface names and limits, reapply all rules)



  • Can i ask if the people in pledged in this bounty consider their offer still valid?!



  • I started this thread about a year ago, but have not been keeping track of pfsense. For my original problem I ended up using m0n0wall and that has worked out well enough. I am not as actively involved in the purchase decision process of the company that needed this solution as I was a year ago. If this feature does eventually get implemented into pfsense I will try my best at ponying up the $200 I had originally pledged. Should anyone need to contact me about receiving my pledge please leave a comment on http://blog.wtip.net/



  • I dont only want to renew my offer, I will raise it again. 400$ for Multi-Interface QoS in combination with PPPoE on OPT-Interfaces. Also I raise my bounty for per-IP Traffic Shaping to 200$.

    Greetings,
    techatdd



  • I'm also highly interested in having traffic shapper running in multiwan, I can offer US$50, I know this is not to much, but is what I can do, and this is for personal use, not for business.

    Saludos
    Alfredo



  • I'll put in 200$ for shaping unrestricted by number of WANs and LANs.



  • Hi,

    I'm throwing in $500 on this one. I'm specifically interested in multiple LAN interface support. Wizard support for this is desired, but not required.

    I wanted to paypal the money right now, but was advised to wait. ;)

    /Eirik


Locked