Traffic shaping usenet 563?
-
G'day fine peoples fans of the best firewall in the world ;D
I am having a slight problem with traffic shaping.
What I want to do
I have a wife (yes :-[) and a usenet download server (separate machine, fixed IP in LAN). I want the usenet downloads to be unlimited speed as long as neither my wife or I am using the internets for different things. If that is the case, I want both wife + me to have unlimited speed, as in: usenet server may even receive zero speed if necessary.(Actually, what I really would want is even wife + me to get same speed, so in an example: my speed down is 20. If wife is nagging me in front of the TV ( ;D), in other words, if neither of us are using the internets from our 'puters, the usenet machine can have the 20. If wife is at her PC and requires 20, she should get 20 and usenet 0. If both wife and me are at our PC's and require the full 20 usenet gets 0 and both wife and me get each 10. But this probably is very complex, so I'll settle for usenet as the lowest priority).
[b]What I did
Upfront: I really have a hard time understanding the 'queues' and 'pipes' and all the different other scary looking field names, and the book didn't help me too much. So I gladly took the wizard ;DI found this thread from Ermal:
http://forum.pfsense.org/index.php?topic=8021.0But I couldn't even add a queue (didn't see the add button), and this user had that same problem it appears:
http://forum.pfsense.org/index.php?topic=12838.0So hence the wizard, especially because of this remark: http://forum.pfsense.org/index.php?topic=47177.0
Run the traffic shaping wizard. About the third or forth page in you will have the option to set different protocols to different priorities - high, normal, and low. Change NNTP to low.
So I used the traffic shaper wizard 'Single WAN multiple LAN' (setting Wan to 1). I used all the defaults (HFSC and such), and indeed, in the last screen, there was an option for NNTP (there is a bug in the GUI, btw: default on top of the page that shaping is enabled, but then all the different options below it, such as NNTP, are greyed out. So you can not select them. I then disabled the flag on the top of the page and then re-enabled it, and then I could select NNTP and set the prioriy to low).
What happened?
It didn't work ;DI simultaneously downloaded a large file from usenet on the download server, and downloaded a PC-BSD iso from my browser, but the browser did not get all the priority. I have attached a screenshot.
Now my hunch is that either one of two things is the problem here:
-
I am assuming that the NNTP in the wizard uses port 119, whereas my usenet is encrypted and hence uses port 563. Perhaps this is the problem?
-
Left and right I have also seen some tutorials where people create firewall rules. I have not done this, as I don't understand exactly what I am doing then ( :-[) and I don't want to break anything. So perhaps this is the reason that it is not sent to the low-priority? But then again, I tried to look in the firewall rules, but I also didn't see the Traffic Shaping Wizard create any rule for the NNTP-119 port, shouldn't it have done that (at least, even though it is the wrong port for encrypted usenet?).[/li]
On that 119-port, by the way, I tried to look around in the Traffic Shaper, tab 'By interface' and 'By queue', but nowhere there do I see 119, so it wasn't as easy as changing 119 into 563 'somewhere'.
Would anybody perhaps be willing to help me? I think I am half way, but I am lost as for what to do next. As always, of course, I will be very grateful for any help ;D
Thank you,
Bye,
-
-
I would use the HFSC scheduler and three main queues: one for the usenet server, one for you an another one for your wife. Set the linkshare value of the usenet server queue to 0%, yours to 50% and your wife's to 50%. This way it should behave pretty much the way you want :)
Of course you have to properly assign traffic to those queues, and it is always a good idea to have a separate queue for ACKs
-
I would use the HFSC scheduler and three main queues: one for the usenet server, one for you an another one for your wife. Set the linkshare value of the usenet server queue to 0%, yours to 50% and your wife's to 50%. This way it should behave pretty much the way you want :)
Of course you have to properly assign traffic to those queues, and it is always a good idea to have a separate queue for ACKs
Thank you for your reply, Georgeman ;D
I managed to get the scheduler working. It was what I suspected but couldn't solve at first (the firewall rules). After quite some googling it turned out I had to look at the Floating rules page, to see that indeed the wizard did create rules there, also for NNTP/usenet. So I changed the rules there from 119 to 563, and the scheduler nicely did move the usenet downloads to the 'low priority' queue that the wizard had created.
(Although: on reboot of the router the shaper started complaining about rules not being there, not being in sync with /temp/… - I don't know I forgot, I just reran the wizard).
As to your suggestion, if I might ask since I am fairly new to all of this and to be honest don't understand too much of it (but I could do your taxes or statutory reporting for your multinational, for that I am qualified ;D :P ;D):
1. Doesn't it mean the usenet gets zero all the time?
2. Given the attached pic: would you mind telling me which queue I would need to copy and customize accordingly, to get to what you suggest?Thank you very much for your help, I do appreciate it :)
-
Simply put, with the HFSC scheduler you have three main parameters:
- Upperlimit, which is a hard cap. The queue won't use more than this bandwidth under any circumstance
- Reatime, which is the guaranteed minimum bandwidth that the queue will have assigned at all times. This means that if the queue is not using the assigned bandwidth, it can be assigned to another queues
- Linkshare, which is the proportional share of bandwidth that the queue will get from the available bandwidth (after all realtime guarantees are met), but it can get more if unused
This might sound confusing, so let's focus on your particular situation (which is a perfect fit for this):
Let's say you have a symmetric 10 Mbps connection. You run some web server or whatever service that needs a guaranteed 40% bandwidth. This service obviously doesn't transfer 4 Mbps continuously all the time, but it might need it at times. Setting a realtime value of 40% means that no matter what, if this service requests bandwidth it will get 4 Mbps. If the service does not use it, it will be available for linkshare.
Linkshare specifies the proportional bandwidth that the queue will get, from the available bandwidth. Let's say you set up three more queues, namely qHollander, qWife and qUsenet, with linkshare values as 40%, 40% and 1% respectively.
If the web server is transferring at 4 Mbps, that leaves 6 Mbps available for the other 3 queues. Then you start a download. You will get the entire 6 Mbps for you, since there's no traffic on the other queues. Then your wife also starts a download. Ok, now the 6 Mbps are splitted in half (since your linkshare value and your wife's is the same), so you will get 3 Mbps each.
Then the Usenet service becomes active. It will get only a tiny fraction of the available 6 Mbps after realtime, let's say 0.1 Mbps, since its linkshare value is really low.
After some time the web server stops, so the full 10 Mbps are available to be "linkshared" among the three queues, in a proportional way (4.9 Mbps for each of you and 0.2 for Usenet, let's say). At some point your downloads finish, so all available bandwidth (10 Mbps) will be used by the Usenet service, unless another queues need bandwidth.Some side notes: in the example I set the linkshare bandwidth as 40/40/1, and as you see that doesn't sum up 100%. This doesn't mean that the remaining 19% won't be used. What you set is actually a proportional relation against the other queues. Setting 3 queues at 10/10/20, 20/20/40, or 25/25/50 will have the same effect (considering there are no other queues). It is probably easier to make them sum up 100% so you can clearly see the assigned bandwidth. Furthermore, the sum cannot exceed 100%. On the other hand, the sum of all realtime value cannot exceed 80% of your total bandwidth.
We could go far beyond with tuning this up by setting the appropriate service curves using the m1 and d parameters (which let you handle the latency), but that is quite more advanced stuff.On your particular setup, just to begin playing around with and to stay simple, I would delete the qOthersHigh and qOthersLow queues, and create three more queues on that same level on both LAN and WAN, as I described earlier. Just assign the required values as both linkshare m2 and bandwidth to avoid confusion (bandwidth is in fact just a shortcut for the linkshare value, best is to make them equal otherwise you won't remember which one really applies).
Then you will have to assign the traffic to the queues via floating rules for example. In this case, the easiest way would be based on the source. Create a rule for example that says "source: you usenet server IP", "destination: any" and assign the qACK/qUsenet queues (for example).Wow, this was longer than expected. Remember that the schedulers are not perfect, anyway. But you should easily notice the shaping taking place.
BTW, when I get to own a multinational I will call you for the taxes and stuff ;)
-
Thank you ex-tre-me-ly very much for the time you have devoted to writing this explanation; I do appreciate it very much :P
I will go and understand it (I hope, but it very well written so even my economists brain should be able to understand it ;D) and try it out. I will report back here.
Thank you ;D
Ah, and yes: once you have incorporated your new multinational, do not hesitate to contact me :P
-
By now I have studied it and indeed, you have a remarkable skill for explaining things very structured and simple; once again thank you ;D
Might I ask one thing which, due to my malfunctioning brain, I didn't quite understand?
On the other hand, the sum of all realtime value cannot exceed 80% of your total bandwidth.
But per your suggestion on setting up the queues, your 'linkshare m2 and bandwith', we don't have realtime, do we? So why the 80%?
Thank you Georgeman ;D
-
Realtime is used to set minimum guaranteed bandwidth. You didn't mention you have anything that needs a guaranteed bandwidth, that's why I invented the "web server" to set the example.
If you do want to set a realtime value for some queues, the sum of those realtime values cannot exceed 80%, but if you don't use it, is fine
You'll see that it is actually useful to further filter down the traffic and use realtime curves so, for example, your downloads or p2p don't clog up your HTTP browsing. Realtime is specially useful for things like ACKs and DNS, which benefit a lot from guaranteed bandwidth
Regards!!
-
Realtime is used to set minimum guaranteed bandwidth. You didn't mention you have anything that needs a guaranteed bandwidth, that's why I invented the "web server" to set the example.
If you do want to set a realtime value for some queues, the sum of those realtime values cannot exceed 80%, but if you don't use it, is fine
You'll see that it is actually useful to further filter down the traffic and use realtime curves so, for example, your downloads or p2p don't clog up your HTTP browsing. Realtime is specially useful for things like ACKs and DNS, which benefit a lot from guaranteed bandwidth
Regards!!
Thank you for your reply, Georgeman: appreciated once again ;D
I think I have expressed myself wrong (that happens a lot with a brain like mine :P ;D): what I didn't quite understand is how you arrive at the 80%(?)
-
The 80% value is per design. The sum of realtime values cannot exceed 80% of the parent's queue bandwidth, otherwise you'll receive errors.
-
The 80% value is per design. The sum of realtime values cannot exceed 80% of the parent's queue bandwidth, otherwise you'll receive errors.
Thank you for your reply once again, Georgeman ;D
After your most excellent explanation I tried to go the route you proposed. But I ran into a problem. I creates queues qWife, qHollander and qUsenet and assigned the 50-50-0 to them. However, Pfsense kept complaining about the child queues being more than 100% of the higher que. I figured this is because qAck also has 20%, so 120% > 100%. But qAck needs to have that, and I found a monst interesting thread (stickied above) about that qAck in my setup actually would need to be 43% on WAN and only 1% on LAN (given 20/2 is a factor 10).
So being the dumb noob that I am I am lost once again; if they can't be 50-50-0, what would you say they need to be?
Thank you for all your help very much ;D
-
Remember that the linkshare value is just a proportional value. If you say that as per your calculation you need 1% of ACK on LAN and 43% on WAN, you can create queues on LAN as:
- qACK: 1%
- qUsenet: 1% (I think it'll complain if you set it to zero)
- qHollander: 48%
- qWife: 48%
On WAN:
qACK: 43%
qUsenet: 1%
qHollander: 28%
qWife: 28%In whatever case, it is a good idea to assign that value you calculated for ACKs as a realtime curve since it is really important for you not to have drops on these queues. If you calculated 43% for ACKs, I would set a little more, let's say 48% as realtime and 5% or whatever as linkshare for the qACK queue on WAN (make sure the sum of linkshare does not exceed 100%)
Regards!
-
Thank you once again for your most excellent, and too kind, help, Georgeman: I am in your debt :P
I tried it again, and you will guess it: it doesn't work :-[
I have created several screenshots and combined them into one that I have attached.
As you can see, the traffic is not assigned to the right que once I have the usenet download server running. Everything stays in the same queue. I am also struggling with the floating rules, and I noticed something strange. The two disabled rules you see in the screenshot are those the wizard generated. The only thing I did was change these rules to change the port to 563. And then it worked. At least traffic came in the right queue due to the port. Already, in these wizard rules, when I added the source-IP (192.168.2.21) and left the ports empty, it didn't work (just as now, with my newly created rule based on your guidance). So that is one problem: the assignment based on SRC IP doesn't appear to work.
The other thing that is strange is that although you can see in the screenshot that the first wizard rule has qACK/qOthersLow, this is [b]not what the rule contains. Because there it only says: qACK/none (screenshot).
Finally, I have attached my floating rule that doesn't work. And I've noticed that the wizard has created two rules for the NNTP-traffic, but I don't know why. I created one with protocol is 'any', this should do the same, shouldn't it?
-
And finally my own firewall rule. Sorry it is slightly difficult to read, as I had to zoom out to get it all on one screen :-X
-
And of course I forgot: thank you very much for helping me out Georgeman ;D
-
I thought perhaps it would be wise to make some screenshots of how I've set up the queues ;D
-
@Hollander:
the assignment based on SRC IP doesn't appear to work.
Yep! And that's right. The reason behind this is that by the time the packet reaches the WAN interface, NAT has already ocurred, so in fact there are no packets with the private IP address to be seen by the rule (it has already been translated).
In order to queue based on the source IP, you need to create the rule on the LAN interface, direction IN. The rest is the same.
As regards the weird rules: sometimes this happens when you modify the queues after the rules were created. Just delete and recreate the rules. Bear in mind that you can only select an acknowledge queue if you also select a regular queue (as the little snippet explains)
-
And once again thank you, thank, thank you, Georgeman ;D
I didn't quite understand this:
The reason behind this is that by the time the packet reaches the WAN interface, NAT has already ocurred, so in fact there are no packets with the private IP address to be seen by the rule (it has already been translated)
But this is incoming (downloading from usenet), so it comes in on the WAN, and after that there needs to be NAT, not before(?)
With regards to this:
In order to queue based on the source IP, you need to create the rule on the LAN interface, direction IN. The rest is the same.
I tried that, but when I want to create a rule on LAN there is no 'match' under 'Action', only block/reject/pass. The floating rules the wizard creates on the other hand have 'Action = match'.
And finally:
Bear in mind that you can only select an acknowledge queue if you also select a regular queue (as the little snippet explains)
I actually don't have a clue why there are two fields there. I do understand what ACK is, but after that I am lost as to why you need to enter two values there, and what they do in the first place. I do have the Pfsense 2.0 book, but unfortunately that doesn't discuss this, and googling for quite some time also doesn't tell me what it is.
I've also send you a PM, btw ;D
Thank you once again Georgeman; the debt keeps on building up ;D
-
The idea behind this traffic shaper is actually to get firewall states tagged as belonging to a specific queue. If you tag an incoming traffic on LAN (request to the internet), then the corresponding traffic incoming on WAN (the actual data download) will be put on the queue with the same name, on WAN, since it belongs to a state already tagged (and this is why the queues on LAN and WAN have the same names).
In fact, the floating rules the firewall creates are actually direction out on interface WAN. So this means that traffic leaving the WAN is what is being matched, not the incoming traffic. Since all traffic incoming on WAN must have been requested from inside, tagging on LAN also works. Also consider that if you do have unsolicited traffic coming in from WAN (a port forward for example), then you need to tag this incoming traffic explicitely on your firewall rules.
On your specific case, what you need to do is create a floating rule, action match interface LAN, direction IN, source IP: your usenet server IP, which assigns the traffic to the appropriate queue.
Does the server accept connections directly from the outside? In that case, you also need to make the NAT rule create an associated filter rule. and then modify that rule on WAN and make it assign the right queue.
On a side note, this same thing could be achieved without floating rules. Pass rules on the interfaces can also assign queues to matching traffic, so you could add a rule on LAN, action pass, above the default allow all rule, with the right queues set and it will be the same thing.
–----------
Regarding the "two queues assignment": the "acknowledge" queue allow you to specify a different queue that will be used for TCP SYN and ACK packets. Usually this is a higher priority queue. This means that for an existing state matching this rule, regular packets will go into the regular queue, and related TCP SYN and ACK packets will go into the acknowledge queue. Of course it does not make sense to specify just the acknowledge queue because if you do so, where the "regular" packets will go?
-
@Hollander:
Thank you once again for your most excellent, and too kind, help, Georgeman: I am in your debt :P
I tried it again, and you will guess it: it doesn't work :-[
[/quote]Hello,
Did you manage to get this finally working ?
I'm interested because I'm only using basic rules for my VoIP … that are working.Tx !
-
@Hollander:
Thank you once again for your most excellent, and too kind, help, Georgeman: I am in your debt :P
I tried it again, and you will guess it: it doesn't work :-[
[/quote]Hello,
Did you manage to get this finally working ?
I'm interested because I'm only using basic rules for my VoIP … that are working.Tx !
What I want not yet, no. But I have reasons to believe shortly it will work. But the default traffic shaping is working (again, I wanted something a little bit different). So you might considering trying the default; simply go through the wizard and raise the priority for VOIP, and then see what happens. Perhaps this works for you too :D