Simple, effective traffic shaping (£50)



  • I will offer £50 to anyone who can implement simple, effective traffic shaping. The system must:

    1. Work with my ISP (NTL 10Mb cable)
    2. Be easy to set up and configure
    3. Be effective

    To elaborate, by "easy" I do not mean it has to have a wizard, but rather that the options it provides must be clear and enough information and feedback given to make choices about configuring them without the need for a degree in networking. Unfortunately, networking was only 10% of my degree :)

    By "effective", I mean it must not just shape traffic but have some appreciable effect beyond simply prioritising ACKs. So, if for example I run a P2P app with no upload cap and 1000+ open connections, web pages should still load quickly and connections not time out. It should also be possible for the P2P app to download at 10Mb (assuming available peers), rather than being crippled.

    NTL broadband is 10Mb down and 512kb up. For some reason, uploading at anything over 11k/sec instantly has a large effect on download speeds. 12k/sec up prevents anything from downloading at over around 300k/sec. I have tested this repeatedly, including by starting an FTP download at a full 10Mb, then starting a rate limited FTP upload and slowly increasing the speed. NTL claim not to traffic shape, except at certain peak times. This effect is repeatable at any time of the day and on other NTL broadband connections, and I have no idea what causes it.



  • We already have just this.  You are most likely having drops in your ACK queues.  Highly suggest visiting the traffic shaping tips and tricks thread and follow the advice in there and then fine tune your queue sizes to prevent drops on the important queues.



  • Thanks, I have already read all the documention and forums posts for the current traffic shaping implementation. I have even posted some questions, but never got a useful response. So, I'm hoping money will do it.

    Even if all that is provided is a configuration that works and clear instructions, I'll pay the £50.

    Oh, and all the graphs show no drops on anything with my current config.



  • Use status -> queues to monitor the queues during the internet spikes.



  • Thanks, I know how to do that. It simply does not work properly, which is why I am offering money to anyone who can make it work the way it is supposed to.

    Maybe the queues display is not working properly? For example, I see zero drops in all queues, yet ping requests time out. It could be my downstream choking (which of course cannot be monitored) but it seems unlikely to be entirely down to that, especially as it happens when uploading only.



  • It DOES work properly.  I have no idea what you are doing wrong but if you jump on irc you will find many happy users that use qos that can help you.



  • It DOES NOT work properly. I have tried IRC before, but I will try again in a moment.

    This is exactly why I posted a bounty. Everyone who replies on the forums just says "it works and you must be doing it wrong." No-one offers an actual solution, they just assume I am an idiot or lying. So, I was hoping some financial incentive would encourage someone to take my request for technical support seriously and actually figure out what is going on.



  • Tell you what, I've got access to a smartbits (http://www.spirent.com/analysis/technology.cfm?media=7&WS=325&SS=110&wt=2) and would be willing to run the tests and further tune the shaper if I can get dedicated hardware to work with.  I'm looking for (hardware) donations for the multi-wan code already, the same gear could be used here.  I can't promise anything other than that I'd work on it - the shaper code needs a serious facelift anyway - but I certainly echo what Scott is saying.  Others have given positive feedback on the shaper, saying that voip calls work fine with full up/downloads, Scott regularly plays  CS:S while uploading new pfSense images.  Personally, I've got enough ISP issues that the shaper buys me nothing (although I still run it), but I can let my torrents seed w/out worrying about voip being affected by my traffic (stupid ISP 10 second latencies I can't fix though - and this happens w/out shaper code and to the modem itself from outside, so it's not pfSense).

    –Bill



  • Interesting offer Bill.

    Is there any way to accurately diagnose ISP issues? I have tried a few free tools but none report any problems. Then again, it wouldn't surprise me if NTL had optimised their set-up to create that illusion.



  • Here's the dumb question that I don't think anyone asked.  Apologies in advance.  If shaping is disabled, do you torrents work properly?  ie. what do the numbers look like given the same test w/out pfSense shaping enabled.

    –Bill



  • If I disable the traffic shaping (or use another router entirely) things still start to get slow with anything over 11k/sec upload. Really, the traffic shaping seems to make almost no difference.



  • @mojo-chan:

    If I disable the traffic shaping (or use another router entirely) things still start to get slow with anything over 11k/sec upload. Really, the traffic shaping seems to make almost no difference.

    Good, now I can blame the ISP and be happy :)  The shaper isn't magic, if the upstream isn't giving you the bandwidth, we can't shape it.

    –Bill



  • It all makes sense now.  Yes, your ISP is causing your problems.  You should offer that 50$ to them for better service.



  • Assume that you really have a 500 kb/s uplink. Set qWANack = 290 kb/s, qP2Pup = 110 kb/s, qWANdef = 100 kb/s, others up queues = 0. Disable any bitorrent upload. Then you can upload (rsync, qWANdef) at 100 kb/s ~ 12 kB/s and still download (bittorrent, qP2Pup) at 7200 kb/s ~ 900 kB/s. I guess this would be the maximum your can reach.



  • Thing is, I can upload via FTP/HTTP/email (sorry I don't use rysnc) at a solid 61k/sec, which shows that it is at least possible to get the full 512k (64k theoretical, so 61k with overheads is pretty good I think).

    Again, NTL claim not to traffic shape. Sure enough, if I set uploads to unlimited with no downloads on BT, I can upload to other people at 60k/sec. Also, I can download over NNTP at a 9.4Mb/sec (not bad considering overheads) so the bandwidth is there. Even if NTL were secretly shaping, I have never heard of anything that limts your download speed after a certain amount of upload bandwidth is used.

    I was wondering if it was some strange thing with NTL or the modem they gave me. Sadly, you have to use an NTL modem and I have their latest (and allegedly greatest) model. It seems okay, only needs rebooting once every three months on average. Maybe it doesn't have enough CPU power to deal with large numbers of outgoing connections, and thus slows down when more than a certain number of outgoing packets are seen per second?

    Could MTU have anything to do with it? NTL say 1500 is the ideal value, which is what I have. I tried 1400 and various steps in between, but it didn't help.



  • Yes. You can upload at nearly 512 kb/s and download at nearly 10 mb/s. But you cannot do both the jobs so fast at the same time. When you're downloading, downlink is used (of course), but uplink is used too. Similarly when you're uploading, not only uplink but also downlink is used.

    So, given your queues are set as noted, you can only download at 7.2 mb/s and upload at 100 kb/s, simultaneously.

    Btw, that calculation was merely idealistic, based on traffic patterns I've observed for MTU = 1500 bytes, which are

    (0.015, 1, 0.04, 0.006) for bittorrent download, and
    (1, 0.024, 0.008, 0.04) for rsync upload.

    Please refer to my recent posts in "Traffic Shaping" box for what the figures mean and how to calculate.

    For other MTU and other services (FTP, HTTP,…), traffic patterns would be different.



  • I understand that I cannot do both at once to the absolute limit, because ACK packets are required.

    However, Traffic Shaping should allow me to set upload speed to unlimited, and make sure ACK packets are prioritised and so get as much bandwidth as required to support whatever download speed I am getting. This does not appear to be happening.

    For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets. Thus, I should be able to set an FTP upload off with no rate cap, and see it get at least 300k/sec with only a minimal (say, 50Kb/sec to account for ACKs etc) effect on downloads. This does not happen, downloads drop to around 250Kb/sec.

    At the very least, the FTP upload should be the thing that is being slowed down by traffic shaping, and downloads should remain fast. That would be enough for me. However, not even that is possible.



  • @mojo-chan:

    For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets.

    I've done a quick-and-dirty NNTP test and seen a notably different pattern.

    Test config:

    • pfsense + pftop. All SSH packets go through qWANdef/qLANdef (I run SSH through an IPsec tunnel).
    • MS Windows 2000 + Outlook Express. Packets go through qOthersUpH, qOthersDownH, qWANack and qLANack.
    • sampling period: 10 second
    • number of samples: 50.

    Observed data (on average):

    qOthersUpH  267.48 B/s
    qOthersDownH  10963.42 B/s
    qWANack  318.78 B/s
    qLANack  88.30 B/s

    That observation corresponds to a traffic pattern of (0.024, 1, 0.029, 0.008). Thus, if I "extrapolate" to 10 mb/s downlink + 500 kb/s uplink, the figures would be:

    qOthersDownH  8929 kb/s
    qLANack  71 kb/s
    total      9000 kb/s

    qOthersUpH  214 kb/s
    qWANack    259 kb/s
    total        473 kb/s

    and there remains very little bandwidth for other uploads.



  • Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me… maybe the queue graphs are wrong?

    Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.

    At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.

    For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.

    I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading. Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.



  • @mojo-chan:

    Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me… maybe the queue graphs are wrong?

    Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.

    Yes, I downloaded a large amount of articles. The packet size in qOthersDownH was about 800-900 bytes.

    Sorry for not making clear that I didn't upload anything through qOthersUpH/qOthersDownH (except NNTP and a very little amount of ICMP and DNS). For NNTP, I tried qOthersUpL/qOthersDownL before, but there're no differences.

    @mojo-chan:

    At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.

    For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.

    Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.

    @mojo-chan:

    I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading.

    m0n0wall can classify packets based on packet size. pfSense can't. So with pfSense one must find it hard to shape, say, bittorrent upload and download.

    @mojo-chan:

    Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.

    That's new to me. I've been believing that linkshare' m2 overrides bandwidth setting…

    It seems you are satisfied with your config. Congratulation. :-)



  • @dusan:

    Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.

    Sorry, I still don't follow you… While I was writing my previous post, BT jumped up to a full 10Mb for a while (couple of minutes). During that time, I was still uploading at 11k/sec, and the total upload bandwidth in use was about 300kb/sec. So, it is possible in real world conditions.

    Are you trying to say that my link should be saturated at 6mb/169kb total (including acks etc)? If that were so, I don't think it would be possible to download at 10mb/sec. I am currently downloading from NNTP with 24 connections at 9.4mb/sec, pfSense showing 280kb/sec for qwanacks and 1.8kb/sec for qwandef. Strange thing is, if I now try to upload at 120kb/sec download speed will drop. The total outgoing would only be 400kb/sec, and traffic shaping is supposed to ensure that ACKs and NNTP control traffic (in qwandef) get priority and so downloads remain unaffected.

    @dusan:

    @mojo-chan:

    Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.

    That's new to me. I've been believing that linkshare' m2 overrides bandwidth setting…

    Maybe it does, frankly I don't know… It was the only parameter not 0 by default, so I tweaked it. Having said that, it didn't seem to make much difference, only bandwidth did.

    I'm still not happy with traffic shaping, and am still offering a bounty to anyone who can get it to work properly. To that end, let me restate a primary goal:

    Allow unlimited uploading in, say, FTP or BT while still being able to download at over 8.5Mb/sec and without dropped connections for HTTP/NNTP/SMTP/POP3 etc (a few BT drops are OK).



  • @mojo-chan:

    @dusan:

    Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.

    Sorry, I still don't follow you… While I was writing my previous post, BT jumped up to a full 10Mb for a while (couple of minutes). During that time, I was still uploading at 11k/sec, and the total upload bandwidth in use was about 300kb/sec. So, it is possible in real world conditions.

    Are you trying to say that my link should be saturated at 6mb/169kb total (including acks etc)? If that were so, I don't think it would be possible to download at 10mb/sec.

    Only uplink is saturated, and 169kb/s was only qP2Pup, not including qWANack. This was only one among infinitely many possible combinations based on a download-only and a BT upload-only pattern. This wasn't accurate, as you didn't show your own patterns (upload-and-download data don't help very much). If you are interested in calculation, try to disable BT download to construct your own upload-only pattern, then disable BT upload for download-only one.

    @mojo-chan:

    I am currently downloading from NNTP with 24 connections at 9.4mb/sec, pfSense showing 280kb/sec for qwanacks…

    9.4 mb/s and 280 kb/s makes sense to me now. In a previous post, your figures said 9 mb/s and 100 kb/s that's why I made a quick test which didn't confirm them.

    Now, let's back to the main problem:

    @mojo-chan:

    …and 1.8kb/sec for qwandef. Strange thing is, if I now try to upload at 120kb/sec download speed will drop. The total outgoing would only be 400kb/sec, and traffic shaping is supposed to ensure that ACKs and NNTP control traffic (in qwandef) get priority and so downloads remain unaffected.

    As you said, and I agree, that 1.8 kb/s are NNTP control packets, which are very important to maintain your download speed. If for some reason, it drops to say, 0.1 kb/s then your 9.4 mb/s download proportionally drops, ie. to 522 kb/s. Control packets have to be priotized, you're absolutely right.

    Unfortunately, they do not get any priority over other packets sent through qWANdef, as packets in the same queue are sent in first-in-first-out order.

    Maybe for NNTP outbound, instead of using qWANdef/qLANdef, you should try qWANack/qLANdef and see if it looks any better. I've never tried this before.



  • snip

    Thanks for that, we are on the same page now I think…

    I'm afraid I can't make download only stats because BT clients limit download speed when you try to upload at less than a certain amount (usually 5kB/sec). I will try seeding (upload only) though, see what I can get.

    @dusan:

    Maybe for NNTP outbound, instead of using qWANdef/qLANdef, you should try qWANack/qLANdef and see if it looks any better. I've never tried this before.

    Ah, so you think the speed problems could be down to the control info being squashed by the massive upload? That sounds like a good theory. All uploads are either ACKs or in qP2Pup, so should be lower priority than the control commands which are in qwandef. Maybe qwandef needs more bandwidth allocated to it? The default from the wizard seems to be totally useless.

    Many thanks for your assistance so far. It looks like it might actually be possible to make traffic shaping work with your ideas on configuring it. I wish someone like you had been around earlier to help develop the wizard or write the shaper documentation.



  • @mojo-chan:

    Many thanks for your assistance so far. It looks like it might actually be possible to make traffic shaping work with your ideas on configuring it. I wish someone like you had been around earlier to help develop the wizard or write the shaper documentation.

    Bleh!  Do you know how much time the authors of the Traffic shaper has spent?  It took 3 iterations to even come up with what we have now….



  • Well, I'm not trying to be harsh… I am a computer programmer (C/assembler/Java mostly) so I know how hard it can be. Actually, it reminds me of writing drivers back in the Amiga days. It becomes very hard to write something good because there is so much variation, so many different configurations to consider. I think that is probably what has happened here.

    There are some really weird things in the TF config that the wizard puts out. For example, making everything default to 1% bandwidth. Maybe it works okay for VOIP or whatever it was tested with, but if you look at the forums a lot of people want it for controlling P2P (especially on shared connections, which mine is).

    Even people like Netgear seem to have trouble implementing TF in a way that works well for everyone. At least with pfSense, the ability to configure it is there. At first I was sceptical about the new shaper code, since it was performing worse than the old ALTQ stuff in m0n0wall. However, it looks like a solution is possible...

    If I get time, I'll try writing up some notes for the wiki. Dusan, do you have a PayPal address?



  • @mojo-chan:

    …since it was performing worse than the old ALTQ stuff in m0n0wall. However, it looks like a solution is possible...

    m0n0wall never used ALTQ. It uses dummynet which is completely different.



  • Mojo-chan, an (unverified) idea is not worth to pay for. Rather, I'm curious about how you  solve the problem. For me, pfSense is an excellent work. It does it job so well that I couldn't reproduce your problem conditions. (I've only experienced bittorrent download drop, but that's an other problem.)

    Perhaps your bounty would be possibly good for the current "Traffic Shaper Changes" project that seems to meet your original requirements. But that's up to you. :-)

    Hope to see your notes on wiki soon.



  • Dusan, I'd be happy to pay you something for your hints. Although you did not provide a full solution like I asked, you at least gave me enough hints to make it work properly. Today, after much fiddling based on your ideas and methods, I managed to have an unlimited BT upload and download at over 8Mb/sec at the same time. Upload speed hovered around 300-330Kb/sec, with ACKs taking the total to around 480Kb/sec which is frankly amazing.

    It's not perfect - you at at least a few high speed connections and not too many uploads (so, for example, eMule and a single HTTP download still isn't right) but generally it is excellent. Web browsing is also very responsive.



  • PS. I added a page to the wiki, last link on the main page. Please take a look.


Log in to reply