Traffic shaper changes [90% completed, please send money to complete bounty]

[SlickNetAaron]

Great! Thank you!  I just sent $75 to Chris.

@ermal:

Yeah it can do multiple level of queues and all of what you describe.

[SlickNetAaron]

So I guess I need to know how to access and install this.  I will get a PM?  This is an embedded install on ALIX.2C3

Regards,
Aaron

@SlickNetAaron:

Great! Thank you!  I just sent $75 to Chris.

[eri--]

@SlickNetAaron:

So I guess I need to know how to access and install this.  I will get a PM?  This is an embedded install on ALIX.2C3

Regards,
Aaron

@SlickNetAaron:

Great! Thank you!  I just sent $75 to Chris.

Yes, pretty soon.

[NateDavis]

If this is the place for tech support questions with the new shapper than great. Otherwise, please direct me where these should go.

I have been playing aorund with the new shapper and either I am really dense, and can't figure it out or I don't understand QoS Properly…  Who Knows...

Anyway, I am trying to prioritize VoIP traffic. This traffic runs over my OpenVPN connection setup in the pfSense. I am having a real problem getting the traffic to register in the voip queue (using the wizard and then modifying the floating tab in rules). Is there anything special I am suppoed to do? I thought about trying to prioritize the openvpn traffic, but couldn't get that to work either. Everything just goes to the default queue.

This is an Avaya ip office setup. I have traffic being tagged with difserv-  DSCP 46, DSCP Mask 63, and SIG DSCP as 0. I tried setting the diffserv in the floating rule to 46, but it still didn't put that traffic in the queue. Any help would be appreciated.

Thanks!
Nate

[hoba]

Shaping inside openvpn tunnels is not yet supported afaik, inside IPSEC should work though.

[eri--]

It is the default LAN rule that is botching it.
Just make it specific or create the rules for the in the LAN tab over the default one supplied by pfSense.
And please try disabling the antilockout rule.

With the new update things should be better(a matter of days since some issues have been fixed).

[SlickNetAaron]

Hi, I don't mean to be impatient.  Just wondering when I may get access to the new shaper.  I can wait for the new update if it is just a couple days.

Regards,
Aaron

@ermal:

With the new update things should be better(a matter of days since some issues have been fixed).

[sullrich]

Should be soon.

[eri--]

For all the bounty contributors.
In the same link as before will find the updated images with several problem fixed.

[eri--]

Get the one with the highest date on it. as -20080324 ;)

[songus]

Ill add 50 to the bounty, should i send them now? when will the image be available¿?

[hoba]

@songus:

Ill add 50 to the bounty, should i send them now? when will the image be available¿?

All bountysupporters get exclusive access to the testingimages and are welcome to testdrive and report back. All others will have to wait for now until there are official builds including the changes. Feel free to send the money in right now.

[eri--]

I explained it a page before:
http://forum.pfsense.org/index.php/topic,2718.180.html

the queue wizard is really a work in progress. the first part is difficult to understand and has text labels in code style. the second part, the one with traffic type prioritization, is an heritage of the old shaper wizard but has no reason to exist, 'cause is not applied anywhere and there's no interface to edit. It seems that now the assignment of traffic type to queues is done within each firewall rule.

Well you do not need any interface to choose since it applies to all interfaces.
Read my explanation of the Floating Tab.

As for the names i will make them more friendly.

BTW, since you are a user what part of the first part you didn't understand?

[k3rmit]

sorry, i just found your 1st explanation, that's why i deleted my post…

i'll try apply the rules as by your tutorial and in case get back to you with a good feedback.

to answer your question, if for example i click on the "single wan multi lan" wizard, i'm asked for the number of connections: in my understanding this should be the LAN and the DMZ, but in the next step i have WAN and OPT1 (DMZ) grouped in the "setup connections speed" section, like if we were talking about two WANs, while the DMZ has to be considered like a LAN section.
i'm puzzled here because given i'm configuring multiple lans, as by wizard name, i should be asked just for the wan bandwidth and then describing the lan part. this could be a limit of my understanding of the shaping mechanism within pf, but i have to admit that the wizard isn't a lot descriptive about what am i doing with the info i'm entering and the options i'm choosing.

i just want to avoid traffic shaping between the LAN and DMZ and meanwhile shape all traffic from all interfaces to WAN: from your tutorial i understand that i just need to assign floating rules to queues. i have a solid heritage of rules assigned to each interface, so i think it will take time to make it work correctly. is there any monitoring/debugging application for pf out there?

btw, thanks for the prompt answer.

[eri--]

Oh for the Multi Lan wizard i might have missed some labels changes.
Though it really asks you for the number of LAN's. As i can not guess what interfaces are considered LAN in your cases.
You see WAN in there since i need to know on which interface is the internet connection connected.

If you do not want to shape traffic between DMZ and LAN, on the traffic shaper config:
1- Click the lan root node on the tree. Set its interface bandwidth to the same as you Network card speed(i.e. 100Mb)
2- Delete the traffic shaper config on both LAN and DMZ
3- Create a queue called qInternet in both the LAN and DMZ interface and setup it with the download speed of your internet connection.
If you have choosen HFSC scheduler make its linkshare m1=m2=link download speed and d =something.
4- Create a DMZ queue on both the LAN and DMZ interface. Setup its bandwidth = Lan root speed - speed of qInternet queue
5- Under the qInternet queue replicate the queues that gets created by the wizard, so that the internet shaping for LAN and DMZ works ok.

Than create a rule that matches local traffic(traffic between LAN and DMZ) and sends it to the qDMZ queue so it does not have limitations from the shaper.

I am testing this setup and will make the changes for the Multi Lan wizard, at least, to produce the above automatically.

You will get it with the next update which fixes the other reported issues.

Just a stupid text illustration of the above is:
WAN
–-qACK
---qDefault
---qP2P
---qVoIP
---qOthersHigh
LAN
---qInternet
----------qACK
----------qDefault
----------qP2P
----------qVoIP
----------qOthersHigh
---qDMZ
DMZ
---qInternet
----------qACK
----------qDefault
----------qP2P
----------qVoIP
----------qOthersHigh
---qDMZ

On the floating rules tab make a rule:
1- pass
2- select LAN and DMZ interface
3- Direction any
4- from any  (though you might consider only the ports to the DMZ services)
5- to any (though you might consider only the ports to the DMZ services)
6- queue qDMZ

And done.

Another more advanced scheme might be:
WAN
---qACK
---qDefault
---qP2P
---qVoIP
---qOthersHigh
LAN
---qInternet
----------qACK
----------qDefault
----------qP2P
----------qVoIP
----------qOthersHigh
---qDMZ
----------qDMZACK
----------qDMZDefault
----------qDMZP2P
----------qDMZVoIP
----------qDMZOthersHigh
DMZ
---qInternet
----------qACK
----------qDefault
----------qP2P
----------qVoIP
----------qOthersHigh
---qDMZ
----------qDMZACK
----------qDMZDefault
----------qDMZP2P
----------qDMZVoIP
----------qDMZOthersHigh

And propper rules in place.

[sullrich]

LANs are easy to determine.  Walk the configuration and look for interfaces without a gateway attached to them.

[SlickNetAaron]

Hi Ermal,

Thanks for allowing access to the new shaper.  I see you are continuing to work on it.

I'm having a very hard time trying to figure out how to set this up.  I am unable to add queues to interfaces (I got it to succeed only once!) I'm totally not understanding how this shaper is laid out - it just does not seem intuitive.

My setup was explained here: http://forum.pfsense.org/index.php/topic,2718.195.html
If you can help me understand how to set this up, I would be grateful.  I would even be willing to write up a HowTo to try to explain the new shaper as well as help form the GUI with you.

Regards,
Aaron

[eri--]

Can you please post full details of your configuration.
Bandwidths you want to use etc so i can give you a config.

The upgrade you have has 3 issues:
1- you cannot add queues other than on the Lan.
EDIT: You cannot add queues that are childs of parent interface other than LAN. But you can add childs of other queues on any interface.
2- The Status->queues is shifted to the right as for a missing line for displaying the header ok.
3- The rrd graphs has a typo which does not allow to propperly view the queues graph
4- Floating rules are generated after per tab interface rules so if you have some rules in the specific interface tabs(wan/lan tab) they will spoil the floating rules.
This are just regressions of backporting from RELENG_1. In the next update they will be ok.

In your case you should not have any problems since you want to add queues only for LAN so you should be OK.

Now from what i see you want something like this.
Create an alias with the host you want to limit.

On the wizard check the Penalty box and add this alias on this step.
Also check the catchall option of it.
You should have a scheme like this after it.

WAN
–-qACK
---qP2P
---qVoIP
---qOthersHigh
---qOthersDefault
---qOthersLow
LAN
---qACK
---qP2P
---qVoIP
---qOthersHigh
---qOthersDefault
---qOthersLow

This should set you on for anything you want.
You limit the customers through the Alias config and no need to tweak the rules.
Also if you want a hard limit for them set the uppelimit of qOthersLow(value m2) to the required limit.

Since of issue 4 you do not need any settings on Wan apart specific things you want to block.
Disable anti lockout rule.
And replicate the LAN default pass in rule to the Floating tab and disable that one(for this upgrade you are running.

That's all you need to share all the bandwidth evenly in your setup. Since you say the AP's are limited to 6Mb that's as simple as it can get with the upper scheme.
You can optimize VoIP rules by converting the rules for VoIP to use DSCP(diffserv code point) instead of port based ones; if you know that they use a specific DSCP mark.

Tell me if this suits you.

The other scheme if you wanted to have the hard limit to 6Mb setuped on the pfSense is:
WAN
---qACK
---qP2P
---qVoIP
---qOthersHigh
---qOthersDefault
---qOthersLow
LAN
---qAP1 (m1=m2=6Mb d=line delay)
------qAP1ACK
------qAP1P2P
------qAP1VoIP
------qAP1OthersHigh
------qAP1OthersDefault
------qAP1OthersLow
---qAP2 (m1=m2=6Mb d=line delay)
------qAP2ACK
------qAP2P2P
------qAP2VoIP
------qAP2OthersHigh
------qAP2OthersDefault
------qAP2OthersLow

or
WAN
---qACK
---qP2P
---qVoIP
---qOthersHigh
---qOthersDefault
---qOthersLow
LAN
------qACK
------qP2P
---qVoIP
------qOthersHigh
------qAP1OthersHigh
------qAP2OthersHigh
---qOthersDefault
------qAP1OthersDefault
------qAP2OthersDefault
---qOthersLow
------qAP1OthersLow
------qAP2OthersLow
On this one set the limits for each AP to the specific queue using upperlimit m2 value. Though i doubt you want their Voip queues to be separate since you want both clients to have seemles VoIP.
The last scheme might give you better results but it is hard to understand for someone not knowing what he is doing.

BTW, if you could gather all my postings about the shaper to something readble and skinned :) i would greatly appriciate. I have not yet found the time to do that.

[mikenl]

I haven't pledged to the original bounty, but i made a contribution of $50,00 USD.
I appreciate the work done on the traffic shaper, and would love to take a look at it.

[SlickNetAaron]

Hi Ermal,

Thanks for taking the time to describe the config.  While when you draw out the queues definitions and it makes mostly perfect sense, but I am having trouble.  The shaper is simply not allowing me to add queues at all!  I push ADD Queue button and fill everything out and nothing shows up!  The other portion is: getting from the shaper wizard to the end outcome is very,very  confusing.  The labels are confusing and the interface needs a lot of help.  I just went back to m0n0wall 1.3b10 to play with their shaper last night.  It is MUCH more intuitive and simple.  As simple as it is, it seems to have more functionality, including the ability to limit per IP bandwidth (in a very weird way, but it says it's easy LoL). I hear m0n0wall also will honor RADIUS bandwidth attributes as well? I do not mean to offend, by any means, I just think your shaper could be simplified and made a lot easier for the end user.

1 other problem - while trying to add the queues, the Service Curve options were always grayed out even after clicking the checkbox to enable the fields.

In the end it seemed that nothing would do what I told it to?

@ermal:

Can you please post full details of your configuration.
Bandwidths you want to use etc so i can give you a config.

that would be great.  Details are below.

WAN: 12mb down / 2mb up (Actually, this is a dyamic WAN.. it will burst up to about 16/2.5, but it is committed to 8/1.  If we could figure out a dynamic rule, that would be amazing! Otherwise, I think just setting 12/2 will work as long as low priority traffic is limited to below the 8/1 mark).  I know several people who are looking for this feature.

Want VNC, SSH, HTTP, ICMP and whatever is customary as higher priority.

As mentioned, there are 2 APs and 1 direct connected router to pfSense.  Each AP can have a total of 5 mb of end-user bandwidth (changed from before). Each AP should be able to burst up to the full 2mb upload speed.  The 5mb of usable bandwidth on the APs is half-duplex.  How do we account for that? (ie, if there is 1mb of upload, then there is only room for 4mb of download.)  There will be traffic  coming over the APs to my servers on the LAN or OPT1 as well.  The other router  attached can have equal priority as the APs for WAN bandwidth.  Of course this needs to be shared.  Identification of which AP or router will have to be by subnet. (10.5.x.y=AP1 and 10.6.x.y=AP2 and and 10.4.x.y=localrouter)

I don't have my OPT1 network figured out yet.  It will basically be for servers and such.  Severs are currently on LAN subnets. OPT1 will need to share upload/download bandwidth on the WAN - at just below HTTP  LAN priority (customers surfing the web should be higher priority, but the catchall rule should be lower priority than the OPT1 servers).

@ermal:

Since of issue 4 you do not need any settings on Wan apart specific things you want to block.
Disable anti lockout rule.
And replicate the LAN default pass in rule to the Floating tab and disable that one(for this upgrade you are running.

I totally don't understand why anti-lockout should be disabled, or what you mean with the LAN rules.

@ermal:

Tell me if this suits you.

The other scheme if you wanted to have the hard limit to 6Mb setuped on the pfSense is:
WAN
–-qACK
---qP2P
---qVoIP
---qOthersHigh
---qOthersDefault
---qOthersLow
LAN
---qAP1 (m1=m2=6Mb d=line delay)
------qAP1ACK
------qAP1P2P
------qAP1VoIP
------qAP1OthersHigh
------qAP1OthersDefault
------qAP1OthersLow
---qAP2 (m1=m2=6Mb d=line delay)
------qAP2ACK
------qAP2P2P
------qAP2VoIP
------qAP2OthersHigh
------qAP2OthersDefault
------qAP2OthersLow

or

The above setup looks exactly how I thought it should look.  (Wasn't sure how the last setup would work, but it makes sense on the surface.) However, I am simply unable to Add these queues in the shaper!  And the queues are confusing to me.  I think I am figuring out that any queues on the LAN interface actually control the UPLOAD to the WAN? And any queues on the WAN control traffic going TO the LANs?  It greatly confuses the matter when we don't want traffic shaped between LANs (interfaces).  How can this be simplified?

@ermal:

BTW, if you could gather all my postings about the shaper to something readble and skinned :) i would greatly appriciate. I have not yet found the time to do that.

I think if I can get a more thorough understanding of the shaper I could write an overview to get people to understand some of the basics myself and others are having difficulty with.  It is sometimes hard to read your descriptions ;)  I'm pretty good at documentation - as long as I have a thorough understanding myself.  Are all of your posts regarding the shaper only in this thread?

Regards,
Aaron