Traffic shaper changes [90% completed, please send money to complete bounty]

[GoldServe]

Full ISO install would be nice. Somehow the full install update file killed my full install. Can not find kernel when it rebooted.

[eri--]

@SlickNetAaron:

so now that we have a mostly functioning shaper GUI, I need to understand the rules.

I need to understand the floating rules concept more.

My understanding is that we can have rule and "flag" those packets matching the rule… then continue to match rules until the Quick option is found.  The concept is great!...  The GUI is not.

1.  The advanced options section does not even have the parameters labeled.  How do we know what to put in these fields?  And I think we are missing options necessary for the shaper?  ie. Where can I find the TCP flags section and the TOS section? These appear to be missing, unless I am blind?  How exactly does the advanced section for flags work?  The advanced section tells us what we can do, but not  what type of parameter is valid here.  What on EARTH is the "Maximum new connections / per second" for, and why does it have a drop-box with numbers of unknown units?

The advanced option is the same as in 1.2 it just has 2 more fields that just mark the packets as in every marking thingy i have used.
To learn more how to use them just see http://cvs.openbsd.org/faq/pf/tagging.html.
The tag = mark in that page.
The tagged = marked.

Since I need my network back and fully operational by Friday afternoon, I need to be able to get these rules in place or else go back to 1.2 release.  These GUI issues must be addressed. I'm not trying to be harsh, but this shaper was claimed to be "90% completed", but in reality it is still in alpha stage.

You do not know what an alpha stage means and do not make silly claims.
I do not think there is anything in there anymore to hold you from creating a working config apart some error checking from preventing the user doing silly things.

BTW, for an interface you need a default queue. It does not mean that for every 'level' you can have a default queue just that for the LAN section you need explicitly ONE AND ONLY ONE DEFAULT QUEUE IN THAT 'LEVEL'.
The same applies to WAN/OPT1 or any other interface. Any of them should only have only one default queue.

ie. Do I just type in "VoIP" for the first unnamed parameter and that is my flag?  And then on the network matching rule, do I use "VoIP" in the 2nd (unnamed) field as the flag that is necessary to match with that particular rule?  Is it possible to use multiple flags in these parameter fields?

So, in my situation, I need to match the traffic type (ack, VoIP, web, ssh, etc).  Then I need to match a Network address to determine which queue to place the packet in.  Then how does that queue pass the packets to it's parent queue?

So, for VoIP(Generic low-delay TOS), what exact rules do I need to flag a packet as "VoIP" and continue down the list of rules.

You have a box labeled DSCP(diffserv codepoint) and you do not need TOS for that if you have DSCP.

So, in my situation, I need to match the traffic type (ack, VoIP, web, ssh, etc).  Then I need to match a Network address to determine which queue to place the packet in.  Then how does that queue pass the packets to it's parent queue?

I do not think this even makes sense but am trying to give you some help.
If you want you can do things as.
1- Floating rule tag packets for VoIP. In the advanced section on the mark input just type "VoIP"
2- On specific interface tab just create a rule that has in the marked input "VoIP" and the specific network you want plus the queue you want it to go to.

These questions are exactly what end users (like ME) are going to be asking.  Why not give the answer before we need to answer 5233 of the same question in the forum?

Thanks for the help.
Aaron

[sai]

@SlickNetAaron:

These questions are exactly what end users (like ME) are going to be asking.  Why not give the answer before we need to answer 5233 of the same question in the forum?

because the dev wil not know what the end user needs help with. docs and howtos are usually written by the enduser.

[SlickNetAaron]

@sai:

because the dev wil not know what the end user needs help with. docs and howtos are usually written by the enduser.

I've already committed to writing the howto.  But not everybody reads directions, either ;)

The dev will know what the end user needs help with by end users (like me) who are not afraid to give good feedback and take significant amounts of time to document the problems so they can be fixed or made idiot-proof before release to the public.

It is my experience that if you can get end users to USE your program or product and you watch how they do things, you can use that insight to make your program more user friendly.  The moment a user says "Huh?" is where a dev should take note and make something more intuitive or at least give a note explaining what we need to do here.  Then you continue on in the process, carefully watching the end user, until they hit the next "huh?" and take note again.

Test and test again until the system flows smoothly for every different kind of situation the users will encounter (as much as possible).  In my experience, the GREAT products out there thrive on getting feedback from users and refining and refining based on that feedback.  The more feedback, the better things become because the dev has the most insight on what people are doing with the product.  Not everyone has an in depth knowledge of BSD, altq, HFSC.  Isn't that the point of making a GUI?

I hope the feedback I offer is appreciated.  Working with other companies to report issues has been GREATLY appreciated because their product can become better as a result.

In this situation, I AM a customer, offering my time and experience to try to help, but it is not well received.

Aaron

[eri--]

Yeah but you make comments on the product by judging it.

If you want to help please tell me what you find unnatural in there and not accuse something does not work only to you it seems counter intuitive.

A firewall is not something simple if you click the advanced options. This try to be addressed with wizards just for this class.

I have attached a picture of what the service curve parameters need to look like when added to the queue.
It works since the first time just it seem that you do not know how to enter them.

Valid values for units in the service curve parameters are (Kb, Mb, Gb, b, %) and they are case sensitive.

You cannot rename a queue cause that means that you want to create a new one. Delete the old and create a new one is not that it is something that difficult afaik.
Furthermore it is not consistent to allow a queue to be renamed cause of the rules that reference it. When you delte a queue it is automatically removed from rules to not break your config.(Just a rule afaik to protect the silly user ;).

P.S. the picture is taken from a debugging output that's why some labels are misaligned.

[SlickNetAaron]

@ermal:

Yeah but you make comments on the product by judging it.

If you want to help please tell me what you find unnatural in there and not accuse something does not work only to you it seems counter intuitive.

I don't try to be a jerk, honest!  I get frustrated, but I just try to give objective information.  I didn't write the errors for my health :)

Valid values for units in the service curve parameters are (Kb, Mb, Gb, b, %) and they are case sensitive.

Yes, I knew these parameters.  I just rebooted a couple times and now queue creation is working just fine with the same values that previously failed.  Hmm??

I am wondering if a lot of my errors are because of the errors I got about not being able to write to the file system and not being able to write to the config?  Maybe this is a bigger problem, outside of the shaper in pfSense or BSD?  Or maybe I have intermittent defective hardware or driver problem for ALIX?

You cannot rename a queue cause that means that you want to create a new one. Delete the old and create a new one is not that it is something that difficult afaik.
Furthermore it is not consistent to allow a queue to be renamed cause of the rules that reference it. When you delte a queue it is automatically removed from rules to not break your config.(Just a rule afaik to protect the silly user ;).

Fair enough.  Just sometimes people make spelling errors.  If the error was on the parent, it is a lot of work to create 10-15 queues again ;)  Something to consider?  I would love to have a "create parent queue" button.

Entering hex values to convert TOS to diffserv: please file this under unnatural :)

I remember the diffserv options used to be listed out like TCP Flag radio buttons were:
Low Delay: yes no don't care
Reliability: yes no don't care

I googled for the hex value equivalent of low-delay TOS  … I can't find a specific value, and I don't have time to become an expert on diffserv.  Where did the radio button go?  Or do I remember it from m0n0?  But for now, can you give me a value that will work for VoIP?  Pretty Please? Plus, I would really love the list of rules the wizard puts out so I can get a config running by this afternoon.

Aaron

[hoba]

@SlickNetAaron:

Entering hex values to convert TOS to diffserv: please file this under unnatural :)

I remember the diffserv options used to be listed out like TCP Flag radio buttons were:
Low Delay: yes no don't care
Reliability: yes no don't care

I googled for the hex value equivalent of low-delay TOS  … I can't find a specific value, and I don't have time to become an expert on diffserv.  Where did the radio button go?  Or do I remember it from m0n0?  But for now, can you give me a value that will work for VoIP?  Pretty Please? Plus, I would really love the list of rules the wizard puts out so I can get a config running by this afternoon.

Aaron

TOS is not DiffServ. Please consult wikipedia if unsure about this and what the difference is between TOS and DiffServ. For TOS you could have these radiobuttons but DiffServ is different. You also can only use either or as both techniques use the same bits in the IP-Header.

http://en.wikipedia.org/wiki/Type_of_Service
http://en.wikipedia.org/wiki/Differentiated_services

m0n0 only supports TOS but DiffServ is superior as it allows more levels of control.

[SlickNetAaron]

@hoba:

TOS is not DiffServ. Please consult wikipedia if unsure about this and what the difference is between TOS and DiffServ. For TOS you could have these radiobuttons but DiffServ is different. You also can only use either or as both techniques use the same bits in the IP-Header.

http://en.wikipedia.org/wiki/Type_of_Service
http://en.wikipedia.org/wiki/Differentiated_services

m0n0 only supports TOS but DiffServ is superior as it allows more levels of control.

Thanks for the clarification.  I'll take a look at that.  Ermal told me to use Diffserv when I asked about TOS. "You have a box labeled DSCP(diffserv codepoint) and you do not need TOS for that if you have DSCP."

The question remains: WHERE and HOW do I set the rule that will identify my VoIP traffic?  I can't find the option for generic Low-Delay TOS and I don't know the DiffServ value.  If I need to enter a DiffServ value, what hex value do I use?  I can't run the shaper wizard or it will destroy my queue sets, and I need to set manual rules anyway. I don't really want to use IP or MAC to identify traffic, but I guess I can in the immediate term.

This is why I have asked a couple times for the detailed list of rules the wizard generates ;)  I can use them as a template.  I know just enough to be dangerous ;)  I know high level stuff, and even some low level things.  But I do not possess the knowledge for getting into bit level details of diffserv and stuff like that.

I just need to create rules to ident traffic… something like this:
High Priority flag: dest 80, 443, 53, 5100, 22, etc
VoIP: low-Delay TOS (or equivalent diffserv) .. or I may have to list individual IPs
catchall (Not sure how to create this one???)

Then rules for each /16 subnet to put the ident flags in the appropriate ack & queue for that subnet.
If this does not sound correct, please let me know.

Thanks for your help!
Aaron

[eri--]

Here is a trick.

Keep your QUEUE config in opt1 interface.
Run the wizard and select only one conection(either multiwan or multilan) follow the wizard and it will create the queues and rules.
After finishing the wizard and having the rules ready to modify you can than go to Firewall->Traffic shaper ->By queues view
Select the opt1 interface from the list and select "copy/clone queues" over the Lan interface and than Wan if you want the same there to.

Now just follow the rules on the floating tab and modify those at your will.

That should keep your config and give you a template of rules.

Is this ok for you?

I cannot give you the template since it is generated in code and they are not hardcoded rules.

[SlickNetAaron]

@ermal:

Here is a trick.
Is this ok for you?
I cannot give you the template since it is generated in code and they are not hardcoded rules.

That sounds like a good trick!  I just ran it like that, but got this error on on the last screen that has "finish" button.

Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Invalid argument supplied for foreach() in /etc/inc/shaper.inc on line 41 Warning: Cannot modify header information - headers already sent by (output started at /etc/inc/shaper.inc:41) in /usr/local/www/wizards/traffic_shaper_wizard.inc on line 535

D'oh!  The end result is this error and these queues (the rules generated, but with only 2 queues)
php: : New alert found: There were error(s) loading the rules: pfctl: should have one default queue on vr1 pfctl: errors in altq config The line in question reads [ should have one default queue on vr1 pfctl]:

since my queues are destroyed now anyway (I did copy them to OPT1..?? But I think I ran the wrong wizard) I'll start over fresh one more time.  crosses fingers

Aaron


[eri--]

could you tell me what options did you choose during the wizard? And which wizard you ran?

[SlickNetAaron]

I believe I ran:
Single Lan multi Wan traffic_shaper_wizard.xml

1 "connection" (again, is this referring to LAN or WAN connections?)
LAN: HFSC
WAN: HFSC, 4mb upload, 16mb download
voip, generic low delay, 512/512
penalize IP: 10.0.0.244, 20% (dummy address just to get the rules for templates)
p2P check, catchall check, 20%
no gaming options
Other network protocols: defaults except I set Higher on: VNC, ARD, PPTP, IPSEC, HTTP, DNS, ICMP

I just ran the wizard again without error.  The difference this time is that I REMOVE SHAPER before running the wizard.  Perhaps it was having problem with the existing queues in the config?  I dunno

Aaron

[eri--]

When you choose multiple Wan wizards it refers to internet connections.

For multi LAN wizards it refers to number of internal networks ie number of local interfaces that will be connected to local networks.

[SlickNetAaron]

@ermal:

When you choose multiple Wan wizards it refers to internet connections.

For multi LAN wizards it refers to number of internal networks ie number of local interfaces that will be connected to local networks.

I know that after multiple runs thru the wizard, but my point is that it's not obvious.  Some wizard(s) do specify Local and WAN, some just say num of connections.

Anyway, more bug time!  ::)

1. The rules page loads VERY slowly and often fails to complete loading.  Attaching a couple screen caps.  Almost every time I have to refresh the list to get a full populate.  Mostly on the floating rules, but WAN is having same issue.  I am guessing this an issue outside of the shaper?

2. Rules created by the wizard

• First thing I noticed is my VoIP goes to my P2p catch all queue.  The voIP rule seems to have UDP as the only identifying portion of the rule?  Where is the TOS/DiffServ flags?  This used to work flawlessly in 1.2RC4

• Do we still need to delete the default LAN rule and create one on the float tab?

• Do we still need to disable the anti-lockout rule?

3. Floating rules interface:
I like the concept of tagging a LOT.  But, I think that mixing the queues in with the firewall rules is confusing. Maybe I'm just not knowledgeable enough, but I am paranoid of the interaction of creating Pass rules in the firewall to use the shaper queues.  I just reloaded 1.2 Release and making shaper rules with targets, TOS & TCP flags just seems a lot more intuitive. Plus, it idents my VoIP correctly.

I have decided that I am going to use 1.2 Release for the time being.  I'll get out of your hair, let you work.

Please, Please, Pretty please… test everything and polish things up before releasing this again. Walk through what a user would do in a few scenarios.  Forget your intimate knowledge of what you created, and try to go thru it like you have never seen it.  Of course test with real traffic to make sure rules are matching (I think absolutely everything is ending up in the catch all queue for me right now! I didn't check the lockout rule tho.) Read each label and try to config using only the directions on the screen.  You will see what I mean.

I look forward to a 1.3 beta where others have tested the shaper and things are working much better.
I'll keep an eye on this thread.  Please feel free to ping me if you would like me to do some testing or get some feedback.

Regards,
Aaron

EDIT: I am still committed to writing a HowTo.. But I'd like thing to be in more final form before I prepare it.

[superwormy]

How much do I need to contribute to get a 1.2 package for this? I can't send much immediately but I could send $50 immediately if it means I can:

1.5mb T1 connection
a) Limit DMZ upload/download to/from WAN to 512kb/sec
b) Not limit DMZ upload/download to/from LAN

Is this possible/is $50 enough to get access to the 1.2 packages?

[k3rmit]

Hi,

i know that maybe i wasn't supposed to do that, but i've downloaded the last update image from the location ermal gave me the last time, named

pfSense-Full-Update-1.2-RELEASE-20080402-1748.tgz

Do not use it!! The kernel doesn't load on my machine after the update, i will try to figure out how to fix that…

albe

[eri--]

k3rmit is this an embedded update?!

Since others have reported to upgrade just fine!

[k3rmit]

sorry to have such few time to follow this thread ermal, i still owe you an answer regarding a shaper error… which is: i managed to disable it, reset the configuration and reconfig again correctly. I suppose something got wierd with the first shaper setup, that subsequently created an interpretation error with the update.

To answer your last question, no, is not embedded and thanks for the new link you sent me, i will have a look at it tomorrow morning (it's midnight here).

cheers

albe

[bogus]

Well, I haven't been around for some time and if I see the postings during my absence it looks like no many people having problems to setup and configure the new shaper.
Sorry, but I do have some difficulties to get it working.
The lastest available update (20080409-1911) does not have the new shaper, or at least to wizard looked like to old one.
So I downloaded 20080402-1748 and applied it to a fresh 1.2-RELEASE installation (downgrading from 20080409-1911 does not worked).
Once finished the basic configuration I moved the "Default LAN rule" to the floating tab and disabled the webGUI anti-lockout rule.
To keep it simple the load-balancing pools have been created but no rules to use them have been created.
Only the floating tab is having one rule.
So far everything good, I still could access to webGUI and the clients could access the internet.

Now I walked through the single LAN Multi WAN Wizard:
numberofconnections: 3
Put in the values of my ADSL connections (still don't know if I should substract the PPPoE overhead? But guess, yes!) and select HFSC scheduler.
Enable Prioritize Voice over IP traffic.
No Penalize IP or Alias.
No Lower priority of Peer-to-Peer traffic (At a later stage I do want this but for now I want it as simple as possible).
No Prioritize network gaming traffic.
Yes Other networking protocols, set HTTP and MSN to higher priority and SMTP. POP3, IMAP and Lotus Notes to Lower priority.
Finish.

The following rules at the floating tab have been created:

Proto  	Source  	Port  	Destination  	Port  	Gateway  	Queue  	Schedule  	Description  	
UDP 	* 	* 	* 	* 	* 	qVoIP 	  	DiffServ/Lowdelay/Upload  	
TCP 	* 	* 	* 	1863 	* 	qACK/qOthersHigh 	  	m_Other MSN1 outbound  	
TCP 	* 	* 	* 	6891 - 6900 	* 	qACK/qOthersHigh 	  	m_Other MSN2 outbound  	
TCP 	* 	* 	* 	6901 	* 	qACK/qOthersHigh 	  	m_Other MSN3 outbound  	
UDP 	* 	* 	* 	6901 	* 	qOthersHigh 	  	m_Other MSN4 outbound  	
TCP 	* 	* 	* 	80 (HTTP) 	* 	qACK/qOthersHigh 	  	m_Other HTTP outbound  	
TCP 	* 	* 	* 	443 (HTTPS) 	* 	qACK/qOthersHigh 	  	m_Other HTTPS outbound  	
TCP 	* 	* 	* 	25 (SMTP) 	* 	qACK/qOthersLow 	  	m_Other SMTP outbound  	
TCP 	* 	* 	* 	110 (POP3) 	* 	qACK/qOthersLow 	  	m_Other POP3 outbound  	
TCP 	* 	* 	* 	143 (IMAP) 	* 	qACK/qOthersLow 	  	m_Other IMAP outbound  	
TCP 	* 	* 	* 	1352 	* 	qACK/qOthersLow 	  	m_Other LotusNotes1 outbound  	
UDP 	* 	* 	* 	1352 	* 	qOthersLow 	  	m_Other LotusNotes2 outbound  	
* 	LAN net 	* 	* 	* 	* 	none 	  	   	 

I would expect that HTTP traffic would go into qOthersHigh and receiving an email (8MB attachment) with Thunderbird into qOthersLow.
OK, the outgoing port is set to 587 because port 25 is blocked here, but the incoming is default on port 110.

But it does not, everything goes into qDefault (WAN and LAN).

Do I need to configure something else?

Cheers

[voona]

Hi guys,

Happy to pledge 50$ to get openvpn tunnels working with the Shaper.. Is this possible? Will it be implemented?

Regards,