Traffic shaper changes [90% completed, please send money to complete bounty]

mikenl

@ermal:

http://cvstrac.pfsense.com/chngview?cn=21849
Found the problem it should not happen only with the Sinlge Lan multi Wan wizard.

If you can't wait for the next build do your fixes accordingly it is not hard afaik.

Fixed it indeed, thanks.

eri--

SlickNetAron i am building it. Check the link i gave i will update it there.

You will notice from the date.

SlickNetAaron

@ermal:

SlickNetAron i am building it. Check the link i gave i will update it there.

You will notice from the date.

I'm not seeing this yet.. Is it the link that has ermal in the url?

Thanks,
Aaron

GoldServe

Can I get a quick who to send money to and where to request access for the embedded 1.2 images?

I just found out that when I torrent, a lot of my ACKs are delayed so surfing web pages is much slower. I just want a simple configuration to prioritize http traffic and ACKs.

Thanks.

SlickNetAaron

FYI, The shaper wizard built into 2.1 release will already do this.

This new shaper isn't quite finished yet (after 1.5 weeks I have yet to get a working copy), and I believe it will be released to the public for free in v1.3.

Unless you have multiple WANs or multiple LANs, it doesn't sound like you need this new shaper? I'm just letting you know that you can already do what you like in the current 1.2 release. If you are running 1.2, you probably do not have it configured correctly.

If you use pfSense and appreciate the work of the devs, please donate! These guys work very hard, and pfSense is pretty great! Just from the info you gave, it doesn't sound like you need the new shaper. Your situation, as described, is quite basic.

Regards,
Aaron

@GoldServe:

Can I get a quick who to send money to and where to request access for the embedded 1.2 images?

I just found out that when I torrent, a lot of my ACKs are delayed so surfing web pages is much slower. I just want a simple configuration to prioritize http traffic and ACKs.

Thanks.

GoldServe

@SlickNetAaron:

FYI, The shaper wizard built into 2.1 release will already do this.

This new shaper isn't quite finished yet (after 1.5 weeks I have yet to get a working copy), and I believe it will be released to the public for free in v1.3.

Unless you have multiple WANs or multiple LANs, it doesn't sound like you need this new shaper? I'm just letting you know that you can already do what you like in the current 1.2 release. If you are running 1.2, you probably do not have it configured correctly.

If you use pfSense and appreciate the work of the devs, please donate! These guys work very hard, and pfSense is pretty great! Just from the info you gave, it doesn't sound like you need the new shaper. Your situation, as described, is quite basic.

Regards,
Aaron

I do have mulitple WANs but it seems my problem may be due to other reasons..like PFsense is overloaded in packets or some tcp window size issue as described here: http://forum.pfsense.org/index.php/topic,8698.0.html

I would still like to donate for all the hard work put into this project and maybe try the new shaper if it can help my problems.

eri--

This new shaper isn't quite finished yet (after 1.5 weeks I have yet to get a working copy), and I believe it will be released to the public for free in v1.3.

Pretty nervous buddy what's wrong?! You think you are at a retal store in here?!

Anyway teh build is running for embedded when it is finished you will find it there.

Please, honor your claims by gathering all this things into a nice tutorial.

mikenl

Found a rather simple writeup of HSFC basics with some nice examples, thought others could benefit from it.
http://calomel.org/pf_hfsc.html

Still trying to find a bit of text explaining why drops happen, and i don't see any borrows,
cause in my understanding right now HFSC is all about scheduling time based more than solely prioritize traffice.

Ermal, if i use the traffic shaping wizard 1 wan multi lan, than in my setup the qLocal queues get assigned a negative bandwidth for some reason and
you'll get errors. No biggie, but i guess i should mention it.

eri--

Yeah i hardcoded a value of 10Mbits in there and i guess i was wrong ::)

What do you mean drop happens and you do not see any borrows?!
With HFSC you will not see any borrows on Status->Queues. That is there only for CBQ scheduler.

You can increase the queue limit somewhat to eliminate the drops.
The default size is 50 slots. If you double it be aware that you might tradeof latency with that.

mikenl

yeah makes perfect sense with the numbers i'm getting.
Ok borrows are for CBQ only.
Then i'm still wondering why i sometimes get 200000+ drops on a queue.
Is the queue not getting enough bandwidth ? or is it thorreteling then ?

ridnhard19

Hi ermal, I hate to bring more bad news, or maybe I dont know how to use your version of the *.img file but it does not work. The file size is almost 4 times the size of a normal embedded image during the RC's and final releases of the normal software. With this said I gave it the benifit of the dought and flashed it; would not boot. I checked to see if it was compressed (part of flashing you decompress and pipe to std in of dd; following instructions) and It was not.

I compressed the file to see if we got something close to the size which is distributed on the website as the standard FINAL release of the last version; it ended up being 380kb. Something does not seem quite right or I dont know how to use your specific IMG file.

If someone pointed me to directions I can try and help roll some of the embedded images and test them before they get pushed out to the masses. This might help eliminate some of the frustration.

Thanks for all your work on this, I have gotten it working, was having trouble with the floating rules but I understand this new version fixes that. Looking forward to seeing it in action again.

@ermal:

This new shaper isn't quite finished yet (after 1.5 weeks I have yet to get a working copy), and I believe it will be released to the public for free in v1.3.

Pretty nervous buddy what's wrong?! You think you are at a retal store in here?!

Anyway teh build is running for embedded when it is finished you will find it there.

Please, honor your claims by gathering all this things into a nice tutorial.

sullrich

Images updated. Try pfSense.img.gz 02-Apr-2008 23:39 24.6M

SlickNetAaron

@sullrich:

Images updated. Try pfSense.img.gz 02-Apr-2008 23:39 24.6M

So we are using the sullrich folder again :)

Thanks so much for fixing the image! I'm booting now at least! Yaay!

Ermal, you can hold me to writing a tutorial/doc for the new shaper. Just one thing I require before doing that: a working shaper ;D

Assuming this build is good to go for the most part, do you foresee any material changes that would affect my doc? Or is this still a work in progress? I would like to work closely with you to make sure my work (and the doc) is correct.

Aaron

SlickNetAaron

Hi ermal,

I just ran the 1 wan, multi LAN wizard. I get the following error at the end:

There were error(s) loading the rules: cannot determine interface bandwidth for vr2, specify an absolute bandwidthaltq not defined on vr2 altq not defined on vr2 /tmp/rules.debug:26: errors in queue definition altq not defined on vr2 /tmp/rules.debug:27: errors in queue definition altq not defined on vr2 /tmp/rules.debug:28: errors in queue definition altq not defined on vr2 /tmp/rules.debug:29: errors in queue definition altq not defined on vr2 /tmp/rules.debug:30: errors in queue definition altq not defined on vr2 /tmp/rules.debug:31: errors in queue definition altq not defined on vr2 /tmp/rules.debug:32: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [26]: queue qInternet on vr2 bandwidth 15Mb hfsc ( red , linkshare (15Mb, 100, 15Mb) , upperlimit 15Mb ) { qACK, qOthersDefault, qP2P, qVoIP, qOthersHigh, qOthersLow } …

I'm going to run with the 1 interface and see how she goes.

Aaron

eri--

That is a problem of the vr driver which has been fixed in the latest versions of it.

What you can do after running the wizard is to set the bandwidth of root queue ie "lan, wam,opt1" if it is not set since the driver does not automatically report its link layer speed to altq.
Usually that would be LAN where i do not set the bandwidth explicitly since create the new structure as
–rootqueue
-----qInternet
------------corresponid queues

I actually cannot do much about that for now. And i have not that kind of hardware to really test for all drivers that do not report this. So just a WARNING to users about that, sorry!

The material want change about the shaper.

Only when new queue disciplines come in. In fact they are addition to the docs to descrbie this new discipline.

One thing to get verified for you that are using this update for 1.2 is the generation of the Floating Rules before the other tabs.

You can do that by checking the /tmp/rules.debug and find the comment
#User defined rules and see if rules of the floating tab are before the other specific interface tab rules.
This is just to be sure they are ok and more eyes on it is not harmful.

I hope there are no more issues in this build/update so you can use it on your environments.

Thank you for your patience and sorry for any noise introduced on your environments.

GoldServe

$50 donations sent to paypal@chrisbuechler.com! Thanks!

SlickNetAaron

Embedded Build: Wed Apr 2 23:31:42 EDT 2008

Sorry Ermal, back to the drawing board!

I was trying to do the setup you explain and I am getting error after error, plus silent failures and successes with crazy error messages.

Specific difficulties/bugs I am experiencing:
Adding a child queue - fails silently unless the first queue is set to default. This is counterintuitive because I was just trying to duplicate the existing queue tree for of the primary Internet queue. Thoughts on resolving: a. making a note for adding the first child queue b. error checking when pressing the save button c. not load the config to hfsc/altq until apply button is pressed. This would allow the USER to input queues in any order they please and minimize frustration.

Editing Queue name: fails silently - name does not change on the queue tree. (I added a queue that had more than 15 character queue name - I got the error and tried to shorten the queue name, but it failed. )
Invalid queue name is not able to delete (caused by the error above)

Attempt to delete parent queue to delete the child queue with invalid Queue Name: error:

Warning: copy(/cf/conf/backup/config-1207241932.xml): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1794 Warning: fopen(/cf/conf/backup/backup.cache): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1801 Warning: fwrite(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1802 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1803 Warning: Cannot modify header information - headers already sent by (output started at /etc/inc/config.inc:1794) in /usr/local/www/firewall_shaper.php on line 82

Attempt to delete child queue off of LAN interface: Error:


Warning: copy(/cf/conf/backup/config-1207241932.xml): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1794 Warning: fopen(/cf/conf/backup/backup.cache): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1801 Warning: fwrite(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1802 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1803 Warning: Cannot modify header information - headers already sent by (output started at /etc/inc/config.inc:1794) in /usr/local/www/firewall_shaper.php on line 82

Added a child queue to qInternet (qAP2Down), then the default child queue (qAP1Default) gives error.

 php: : There were error(s) loading the rules: pfctl: should have one default queue on vr0 pfctl: errors in altq config - The line in question reads [ should have one default queue on vr0 pfctl]:

Added qAP1Ack (child of AP2Down) set priority 7, with no service curve. Error:

php: : There were error(s) loading the rules: pfctl: the sum of the child bandwidth higher than parent "qAP2Down" pfctl: linkshare sc exceeds parent's sc /tmp/rules.debug:33: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ the sum of the child bandwidth higher than parent "qAP2Down" pfctl]:

Why would a blank linkshare say it is greater than it's parent?

Starting over with the shaper…..

Wizard: " numberofconnections: Number of connections you have" Can we please specify if this is for LAN or WAN?
Wizard: "conn0interface:" ????? Who exactly is conn0interface?

At this point, I removed the shaper and started the wizard again. Attempting to create:
LAN
---qInternet
-----qAP1
--------qACK
--------qDefault
--------etc
-----qAP2
--------qack, etc
-----qLocal
--------qack, etc

Deleting existing children of qInternet - happened every time I deleted. However, the shaper GUI does update and appear to delete the queue.


Warning: fopen(/cf/conf/backup/backup.cache): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1801 Warning: fwrite(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1802 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1803 Warning: Cannot modify header information - headers already sent by (output started at /etc/inc/config.inc:1801) in /usr/local/www/firewall_shaper.php on line 82 

Warning: copy(/cf/conf/backup/config-1207246821.xml): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1794 Warning: fopen(/cf/conf/backup/backup.cache): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1801 Warning: fwrite(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1802 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/config.inc on line 1803 Warning: Cannot modify header information - headers already sent by (output started at /etc/inc/config.inc:1794) in /usr/local/www/firewall_shaper.php on line 82

How does the LinkShare work inside of a Child queue? If I write 5% linkshare, is that 5% of the direct parent, or the root?

When I get to:
lan
–qInternet
----qP2P default (can't modify or delete!)
----qAP1 default (cannot remove default flag)
-------qAP1CatchAll(Default)

I get the error below. I try to add and delete any possible combinations of Default Flags on the tree of queues. It just won't accept the config. Also, the changes to Default Queue flags seem to save (silent failure), but they revert back and appear to not take effect.


: php: : There were error(s) loading the rules: pfctl: should have one default queue on vr0 pfctl: errors in altq config - The line in question reads [ should have one default queue on vr0 pfctl]:

Attempting to start from scratch: no wizard. My exact actions are as follows:
LAN: bw: 100mb
SAVE, Add New Queue
15mB, qInternet, Priority 6
UpperLimit: 15Mb 30000 8Mb
Save, Apply


Error: php: : There were error(s) loading the rules: pfctl: should have one default queue on vr0 pfctl: errors in altq config - The line in question reads [ should have one default queue on vr0 pfctl]:

Add default Flag to qInternet
Save, Apply, no error
ADD Queue button is not available to add child queue to qInternet???
Delete qInternet to try to start over.
Apply.
Interface with no label appears above lan, Queue not found error box displayed (attachment: shaper-phantom interface.jpg}
)

Clicking on the phanton interface yields the Queue not found error again. The only way to remove it is to use "Remove Shaper" button.

Going to try again without using qInternet parent. (This will NOT work for my setup, but going to see if it is a problem with multiple layers of queues.)
Again, my exact actions:
Click on LAN interface
15MB bandwidth is pre-populated
save & Apply
Add New Queue
Bandwidth 5Mb
qAP1
Default
upperlimit m2= 5mb
save & apply
I want to add a child queue to this, but the Add New Queue button and Delete buttons are gone.

Tell me what I did wrong?! I'm following the steps exactly as one is able to do given the GUI provided. It just feels so much like the end user (a human, NOT a programmer) is being forced to think like a programmer and if we don't do exact steps it fails horribly. Why can't the user get things setup the way they want and let the programming handle the details? That is what a good GUI is all about. I feel like I am constantly having to fight the GUI to get to do what I want… the GUI always wins and I lose and don't end up with a working shaper.

Is it possible to write the darn config manually??? I am pretty sure I know what I need things to look like.

Aaron

![shaper-phantom interface.jpg](/public/imported_attachments/1/shaper-phantom interface.jpg)
![shaper-phantom interface.jpg_thumb](/public/imported_attachments/1/shaper-phantom interface.jpg_thumb)

eri--

Can i have access to your box.

I didn't understand much of your posting but i am not able to replicate some of your errors here.

Or please after you find the error send me config.xml and rules.debug?!

eri--

Attached just a demonstartion of what the can be done with the new shaper as for queue creation.

crzay_queues.png_thumb

SlickNetAaron

@ermal:

Attached just a demonstartion of what the can be done with the new shaper as for queue creation.

Yup, it's true! I think he took that screen cap from my box ;)

Thanks for the help. I think it's working now! There are some bugs, but it's workable.

Aaron