Simple, effective traffic shaping (£50)
-
I will offer £50 to anyone who can implement simple, effective traffic shaping. The system must:
1. Work with my ISP (NTL 10Mb cable)
2. Be easy to set up and configure
3. Be effectiveTo elaborate, by "easy" I do not mean it has to have a wizard, but rather that the options it provides must be clear and enough information and feedback given to make choices about configuring them without the need for a degree in networking. Unfortunately, networking was only 10% of my degree :)
By "effective", I mean it must not just shape traffic but have some appreciable effect beyond simply prioritising ACKs. So, if for example I run a P2P app with no upload cap and 1000+ open connections, web pages should still load quickly and connections not time out. It should also be possible for the P2P app to download at 10Mb (assuming available peers), rather than being crippled.
NTL broadband is 10Mb down and 512kb up. For some reason, uploading at anything over 11k/sec instantly has a large effect on download speeds. 12k/sec up prevents anything from downloading at over around 300k/sec. I have tested this repeatedly, including by starting an FTP download at a full 10Mb, then starting a rate limited FTP upload and slowly increasing the speed. NTL claim not to traffic shape, except at certain peak times. This effect is repeatable at any time of the day and on other NTL broadband connections, and I have no idea what causes it.
-
We already have just this. You are most likely having drops in your ACK queues. Highly suggest visiting the traffic shaping tips and tricks thread and follow the advice in there and then fine tune your queue sizes to prevent drops on the important queues.
-
Thanks, I have already read all the documention and forums posts for the current traffic shaping implementation. I have even posted some questions, but never got a useful response. So, I'm hoping money will do it.
Even if all that is provided is a configuration that works and clear instructions, I'll pay the £50.
Oh, and all the graphs show no drops on anything with my current config.
-
Use status -> queues to monitor the queues during the internet spikes.
-
Thanks, I know how to do that. It simply does not work properly, which is why I am offering money to anyone who can make it work the way it is supposed to.
Maybe the queues display is not working properly? For example, I see zero drops in all queues, yet ping requests time out. It could be my downstream choking (which of course cannot be monitored) but it seems unlikely to be entirely down to that, especially as it happens when uploading only.
-
It DOES work properly. I have no idea what you are doing wrong but if you jump on irc you will find many happy users that use qos that can help you.
-
It DOES NOT work properly. I have tried IRC before, but I will try again in a moment.
This is exactly why I posted a bounty. Everyone who replies on the forums just says "it works and you must be doing it wrong." No-one offers an actual solution, they just assume I am an idiot or lying. So, I was hoping some financial incentive would encourage someone to take my request for technical support seriously and actually figure out what is going on.
-
Tell you what, I've got access to a smartbits (http://www.spirent.com/analysis/technology.cfm?media=7&WS=325&SS=110&wt=2) and would be willing to run the tests and further tune the shaper if I can get dedicated hardware to work with. I'm looking for (hardware) donations for the multi-wan code already, the same gear could be used here. I can't promise anything other than that I'd work on it - the shaper code needs a serious facelift anyway - but I certainly echo what Scott is saying. Others have given positive feedback on the shaper, saying that voip calls work fine with full up/downloads, Scott regularly plays CS:S while uploading new pfSense images. Personally, I've got enough ISP issues that the shaper buys me nothing (although I still run it), but I can let my torrents seed w/out worrying about voip being affected by my traffic (stupid ISP 10 second latencies I can't fix though - and this happens w/out shaper code and to the modem itself from outside, so it's not pfSense).
–Bill
-
Interesting offer Bill.
Is there any way to accurately diagnose ISP issues? I have tried a few free tools but none report any problems. Then again, it wouldn't surprise me if NTL had optimised their set-up to create that illusion.
-
Here's the dumb question that I don't think anyone asked. Apologies in advance. If shaping is disabled, do you torrents work properly? ie. what do the numbers look like given the same test w/out pfSense shaping enabled.
–Bill
-
If I disable the traffic shaping (or use another router entirely) things still start to get slow with anything over 11k/sec upload. Really, the traffic shaping seems to make almost no difference.
-
If I disable the traffic shaping (or use another router entirely) things still start to get slow with anything over 11k/sec upload. Really, the traffic shaping seems to make almost no difference.
Good, now I can blame the ISP and be happy :) The shaper isn't magic, if the upstream isn't giving you the bandwidth, we can't shape it.
–Bill
-
It all makes sense now. Yes, your ISP is causing your problems. You should offer that 50$ to them for better service.
-
Assume that you really have a 500 kb/s uplink. Set qWANack = 290 kb/s, qP2Pup = 110 kb/s, qWANdef = 100 kb/s, others up queues = 0. Disable any bitorrent upload. Then you can upload (rsync, qWANdef) at 100 kb/s ~ 12 kB/s and still download (bittorrent, qP2Pup) at 7200 kb/s ~ 900 kB/s. I guess this would be the maximum your can reach.
-
Thing is, I can upload via FTP/HTTP/email (sorry I don't use rysnc) at a solid 61k/sec, which shows that it is at least possible to get the full 512k (64k theoretical, so 61k with overheads is pretty good I think).
Again, NTL claim not to traffic shape. Sure enough, if I set uploads to unlimited with no downloads on BT, I can upload to other people at 60k/sec. Also, I can download over NNTP at a 9.4Mb/sec (not bad considering overheads) so the bandwidth is there. Even if NTL were secretly shaping, I have never heard of anything that limts your download speed after a certain amount of upload bandwidth is used.
I was wondering if it was some strange thing with NTL or the modem they gave me. Sadly, you have to use an NTL modem and I have their latest (and allegedly greatest) model. It seems okay, only needs rebooting once every three months on average. Maybe it doesn't have enough CPU power to deal with large numbers of outgoing connections, and thus slows down when more than a certain number of outgoing packets are seen per second?
Could MTU have anything to do with it? NTL say 1500 is the ideal value, which is what I have. I tried 1400 and various steps in between, but it didn't help.
-
Yes. You can upload at nearly 512 kb/s and download at nearly 10 mb/s. But you cannot do both the jobs so fast at the same time. When you're downloading, downlink is used (of course), but uplink is used too. Similarly when you're uploading, not only uplink but also downlink is used.
So, given your queues are set as noted, you can only download at 7.2 mb/s and upload at 100 kb/s, simultaneously.
Btw, that calculation was merely idealistic, based on traffic patterns I've observed for MTU = 1500 bytes, which are
(0.015, 1, 0.04, 0.006) for bittorrent download, and
(1, 0.024, 0.008, 0.04) for rsync upload.Please refer to my recent posts in "Traffic Shaping" box for what the figures mean and how to calculate.
For other MTU and other services (FTP, HTTP,…), traffic patterns would be different.
-
I understand that I cannot do both at once to the absolute limit, because ACK packets are required.
However, Traffic Shaping should allow me to set upload speed to unlimited, and make sure ACK packets are prioritised and so get as much bandwidth as required to support whatever download speed I am getting. This does not appear to be happening.
For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets. Thus, I should be able to set an FTP upload off with no rate cap, and see it get at least 300k/sec with only a minimal (say, 50Kb/sec to account for ACKs etc) effect on downloads. This does not happen, downloads drop to around 250Kb/sec.
At the very least, the FTP upload should be the thing that is being slowed down by traffic shaping, and downloads should remain fast. That would be enough for me. However, not even that is possible.
-
For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets.
I've done a quick-and-dirty NNTP test and seen a notably different pattern.
Test config:
- pfsense + pftop. All SSH packets go through qWANdef/qLANdef (I run SSH through an IPsec tunnel).
- MS Windows 2000 + Outlook Express. Packets go through qOthersUpH, qOthersDownH, qWANack and qLANack.
- sampling period: 10 second
- number of samples: 50.
Observed data (on average):
qOthersUpH 267.48 B/s
qOthersDownH 10963.42 B/s
qWANack 318.78 B/s
qLANack 88.30 B/sThat observation corresponds to a traffic pattern of (0.024, 1, 0.029, 0.008). Thus, if I "extrapolate" to 10 mb/s downlink + 500 kb/s uplink, the figures would be:
qOthersDownH 8929 kb/s
qLANack 71 kb/s
total 9000 kb/sqOthersUpH 214 kb/s
qWANack 259 kb/s
total 473 kb/sand there remains very little bandwidth for other uploads.
-
Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me… maybe the queue graphs are wrong?
Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.
At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.
For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.
I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading. Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.
-
Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me… maybe the queue graphs are wrong?
Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.
Yes, I downloaded a large amount of articles. The packet size in qOthersDownH was about 800-900 bytes.
Sorry for not making clear that I didn't upload anything through qOthersUpH/qOthersDownH (except NNTP and a very little amount of ICMP and DNS). For NNTP, I tried qOthersUpL/qOthersDownL before, but there're no differences.
At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.
For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.
Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.
I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading.
m0n0wall can classify packets based on packet size. pfSense can't. So with pfSense one must find it hard to shape, say, bittorrent upload and download.
Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.
That's new to me. I've been believing that linkshare' m2 overrides bandwidth setting…
It seems you are satisfied with your config. Congratulation. :-)