Traffic Shaping Performance issues

Harvy66

Shaping is bloody simple, but everything you read out there makes it seem hard for some reason. I'll see about creating some screenshots some time this weekend.

There are only four things you need to do

HSFC

Set your interface bandwidth
Create your queues
Set your queue minimum bandwidths
Assign traffic to your queues

#1 seems to be the #1 reason for issue. I have no idea how this part is so hard. If you only have 10Mb of bandwidth, set your Interface to only have 9Mb.
#2 seems to be the next issue. Stop creating complicated hierarchical queues, just create 3, high normal and idle
#3 I can't explain this, same as #1. Stop thinking about priorities and blah blah blah. How much minimum bandwidth do you want for this queue
#4 The wizard does create these for you. Unlike normal firewall rules, for floating, the last one wins

P.S. Don't use realtime, it's confusing, seems to have little benefit in most cases, and is easy to mess up

-flo- 0

@Harvy66:

Set your interface bandwidth
[…]
#1 seems to be the #1 reason for issue. I have no idea how this part is so hard.

I do.

I worked quite a while on this until I found out that the selection 'kbit/s' in the bandwidth setting was plain wrong. It should have been 'bit/s'. (Don't know whether this has been fixed in 2.3 but the bug is still in 2.2.6.)

If as a consequence of this bug I set the bandwidth limit to effectively 10 bit/s (intending it to be 10 kbit/s) the performance was of course completely off …

-flo-

Harvy66

Here is my setup. I'm only showing the WAN because the LAN is mostly identical, just slightly lower interface bandwidth. Some of my settings, like queue size when using CoDel, is probably not correct, and I have funny percentages, but this is exactly what I use. Take it, leave it, learn from it. Do whatever.

WAN.png_thumb

WAN-qACK.png_thumb

WAN-qClassified-Parent.png_thumb

WAN-qClassified-qHigh.png_thumb

WAN-qClassified-qNormal.png_thumb

WAN-qUnclassified-Parent.png_thumb

WAN-qUnclassified-qDefault.png_thumb

WAN-qUnclassified-qUDP.png_thumb

Nullity

@pf3000:

@-flo-:

I worked quite a while on this until I found out that the selection 'kbit/s' in the bandwidth setting was plain wrong. It should have been 'bit/s'. (Don't know whether this has been fixed in 2.3 but the bug is still in 2.2.6.)
If as a consequence of this bug I set the bandwidth limit to effectively 10 bit/s (intending it to be 10 kbit/s) the performance was of course completely off …

-flo-

Thanks for pointing that out. So it's actually bit labeled as Kbit huh. I was confused when the speed test showed 10% of my actual download after setting in Kbit. I settled for Mbit and it's fine. However setting up bandwidth in Kbit via the wizard, it's okay. After that editing to Mbit and back to Kbit messes it up for me. It's like that on both 2.3 and 2.3.1

Hmm… I have used pfSense since 2.1.x and kbit has always been kbit. With 2.3 I am using kbit with no problems.

I extensively tested 2.2.x using mostly "kbit" and never experienced this bug. Are we sure it even exists? Has it been submitted to the pfSense bug tracker?

-flo- 0

@Nullity:

I extensively tested 2.2.x using mostly "kbit" and never experienced this bug. Are we sure it even exists?

I already reported this nearly one year ago, see this thread: Noob guide to Traffic Shaping. You already commented on that thread back then.

At least one user confirmed this problem, others didn't.

@Nullity:

Has it been submitted to the pfSense bug tracker?

I didn't, should I?

-flo-

Nullity

@-flo-:

@Nullity:

Has it been submitted to the pfSense bug tracker?

I didn't, should I?

-flo-

Absolutely.

You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

It might just be easier to post the info here and someone more experienced will submit a bug.

pf3000

It's my mistake; I couldn't duplicate the issue. I knew that traffic wizard always worked, and my internet upload speed is 20% of my download.

While editing the shaper, I may have plugged in up/down bandwidth numbers in lan/wan instead of the other way round.
I also had "Disable hardware large receive offload" unchecked - it hinders my real speed test results.

Hence I got a speed test result of about 10% of the download speed, leading me to think that kbits might be bits. Sorry for the confusion : D

mattlach

@Nullity:

The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
Set bandwidth on WAN interface to ~97% of upstream bandwidth
Set bandwidth on LAN interface to ~97% of downstream bandwidth.

Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

I may have to try this.

It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good. That's where I checked before I posted this thread, but it was lacking. Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question. I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.

-flo- 0

@Nullity:

You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

I worked on this but right now I cannot reproduce this anymore. Originally the results were differing form the configured values by about factor 10. This is not the case in my setup anymore now.

I'll keep an eye on this.

-flo-

Nullity

@-flo-:

@Nullity:

You would need to share some detailed, repeatable steps that consistently show that "kbit" is actually "bit", then someone will either confirm the bug (and fix it) or have further questions.

I worked on this but right now I cannot reproduce this anymore. Originally the results were differing form the configured values by about factor 10. This is not the case in my setup anymore now.

I'll keep an eye on this.

-flo-

Always reset the states, or even just restart pfSense, when playing with traffic-shaping.

Personally, my first few months playing with QoS & pfSense included lots of "why is it doing that?!" situations. Like you, I even thought that the bit/kbit/mbit might be inaccurate. Though, after lots reading & testing, eventually my configurations started to perform as I predicted.

Nullity

@mattlach:

@Nullity:

The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
Set bandwidth on WAN interface to ~97% of upstream bandwidth
Set bandwidth on LAN interface to ~97% of downstream bandwidth.

Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

I may have to try this.

It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good. That's where I checked before I posted this thread, but it was lacking. Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question. I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.

You have it pretty right. My upload connection is ~760Kbit link-rate, with a real-world bitrate of ~666Kbit, so I set my queue to 640Kbit. My ADSL is very stable, so 640 works well enough.

CODELQ is dirt simple, but yeah, there is no fair queueing. You can use either HFSC (Hierarchical Fair Service Curve) or FAIRQ and select the "CoDel Active Queue" check-box to get both fair queueing & codel.

Yeah… searching the forums can be annoying, but you gotta do it. We (pfSense forumites) are not very responsive when it's obvious that a poster has not searched for an answer.

mattlach

@Nullity:

@mattlach:

@Nullity:

The answers to your questions are out there. Searching this forum & Google can answer everything you are asking.

Sorry for the callousness, but I spent ~9 months researching traffic-shaping (mostly relating to pfSense) and I know for a fact that you can find the answers if you search & read. If, after exhaustively researching traffic-shaping, you still have an unsolved problem, please ask. I (and very likely others) will be interested in solving your problem.

but otherwise, much smarter & more educated people have already answered your questions. You simply need to find those answers. ermal's posts in this forum are a favorite resource of mine, but most of his topics may be too advanced. Easy solutions are rare. :)

Well, I did do some searching, and I THINK I've come up with the following, but allow me to bounce it off of you guys to amke sure:

In the "by interface" section under Traffic Shaping set both LAN and WAN to codelq
Set bandwidth on WAN interface to ~97% of upstream bandwidth
Set bandwidth on LAN interface to ~97% of downstream bandwidth.

Now, I also read something interesting about maybe seeing better results of fair distribution of bandwidth (because apparently codel does this poorly) by setting the primary to fairq, and using codel in a child queue.

I may have to try this.

It would seem making edits to the pfsense Traffic Shaping wiki page with some of this information would be good. That's where I checked before I posted this thread, but it was lacking. Peoples eyes glaze over if they have to read through many-page forum threads before finding that needle in a haystack that answers their question. I'd do it myself, but I'm not convinced I have a complete enough understanding of how it works yet to do so.

You have it pretty right. My upload connection is ~760Kbit link-rate, with a real-world bitrate of ~666Kbit, so I set my queue to 640Kbit. My ADSL is very stable, so 640 works well enough.

CODELQ is dirt simple, but yeah, there is no fair queueing. You can use either HFSC (Hierarchical Fair Service Curve) or FAIRQ and select the "CoDel Active Queue" check-box to get both fair queueing & codel.

Yeah… searching the forums can be annoying, but you gotta do it. We (pfSense forumites) are not very responsive when it's obvious that a poster has not searched for an answer.

Hmm, so I finally got around to working on this today.

I did several speed tests in a row to establish downstream at about 152Mb/s and upstream at about 161Mb/s.

Here is how I set it up: (click for full size)

After which I saved all my settings, and made sure to reload the firewall rules.

Then, just as a sanity check I ran another speed test:

This suggests that it isn't working, as it is allowing this single process to exceed the bandwidth limits (the 148 and 156 respectively)

Any idea what I might be doing wrong?

Thanks,
Matt

Nullity

Did you reset the firewall states?

mattlach

@Nullity:

Did you reset the firewall states?

Yeah, the box at the top of the window after you apply the settings tells you to click here to reload the firewall rules in order for it to become effective. I did that.

Is there anything in addition to that I should be doing?

Nullity

@mattlach:

@Nullity:

Did you reset the firewall states?

Yeah, the box at the top of the window after you apply the settings tells you to click here to reload the firewall rules in order for it to become effective. I did that.

Is there anything in addition to that I should be doing?

Try a temporary bitrate (like 20Mbit) that will not be as easily confused with your internet connection's bitrate. The official pfSense wiki's data should work.

mattlach

@Nullity:

Try a temporary bitrate (like 20Mbit) that will not be as easily confused with your internet connection's bitrate. The official pfSense wiki's data should work.

Good idea, I'll try that, thank you.

Meanwhile, could these alerts be related?

Error occurred

SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:45
SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:46
SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:47
SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:48
SHAPER: no default queue specified for interface wan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:49
SHAPER: no default queue specified for interface lan. The interface queue will be enforced as default.	@ 2016-05-02 04:09:50

Harvy66

You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.

mattlach

@Harvy66:

You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.

Yeah, it looks like that might have been it. I selected to make the child queues above the default, and I saw an effect. Unfortunately it was more of an effect than I was hoping to see. With this configuration I was down to 138/150mbit.

At first I thought the fairq interface queue might be to blame, but when I deleted the child queues and set the interface queues to "codelq" i had the same performance loss. Maybe this is how it works? I'm not sure I want to give up THAT much bandwidth though.

Nullity

@mattlach:

@Harvy66:

You need to set one of your queues to be the default. All states must be put into a queue, if you don't supply one, PFSense will, and you won't be able to control it.

Yeah, it looks like that might have been it. I selected to make the child queues above the default, and I saw an effect. Unfortunately it was more of an effect than I was hoping to see. With this configuration I was down to 138/150mbit.

At first I thought the fairq interface queue might be to blame, but when I deleted the child queues and set the interface queues to "codelq" i had the same performance loss. Maybe this is how it works? I'm not sure I want to give up THAT much bandwidth though.

The interface bitrate needs to be lower than your real-world maximum, so tune it as close as possible.

Harvy66

You're set to 148Mb and getting 138Mb. 93% of your total bandwidth is pretty close. In theory, you cannot go above 148, so your average must be below it. The real question is what your bufferbloat is at. Check dslreports.com's speedtest. You can always increase your rates to see how it affects your traffic. The more stable your internet connection, the closer to 100% you can get without getting blips of latency or packetloss.