FLAWLESS VoIP w/ Saturated Download - No Wasted Bandwidth!!
-
We all know the problem. Saturated download wreaks havoc on a lot of things, especially VoIP. You can aggressively limit TCP to the point where voice sounds great, even in the face of saturated downlink - but the cost is a lot of permanently wasted bandwidth.
Using pfSense and a few other tools (tcpdump, Snort, Simple Event Correlator, rate-limiting router), I've figured out a way around that wasted bandwidth. By introducing aggressive rate limiting only while on a VoIP call, I can guarantee excellent voice quality. The only wasted bandwidth, per-se, is strictly for the duration of the call. Once the call is complete, full bandwidth access is automatically restored. I call it Application Aware Triggered Quality of Service (AATQoS).
I've written a complete how-to with instructions, config files, Snort rules, telnet scripts, etc. here: http://www.xmission.com/~hidden/aatqos
I've been using this on my pfSense 1.2-RC3 (and now RC4) setup for the last week and it works perfectly. Comments are welcome and feedback is appreciated.
I'm sure there's a way to do this 100% within pfSense (and not needing a rate-limiting router) - but I'll need help in figuring out which file(s) to modify and how to restart/reload the shaper via the CLI.
Any idea if the "new" shaper will have capabilities similar to this?
-
This is over complicated! (my opinion anyway)
On the shaper even now if you do setup the qVoIP with HFSC scheduler and set realtime to something like this(values depend on your choice, i am setting them with percentage regarding they parent):
m1 = 25%
d = 30
m2 = 20%should keep your quality to acceptable levels.
Dynamic allocation is being discussed http://forum.pfsense.org/index.php/topic,7406.0.html
and here http://forum.pfsense.org/index.php/topic,7502.0.html -
@eri--:
This is over complicated! (my opinion anyway)
On the shaper even now if you do setup the qVoIP with HFSC scheduler and set realtime to something like this(values depend on your choice, i am setting them with percentage regarding they parent):
m1 = 25%
d = 30
m2 = 20%should keep your quality to acceptable levels.
I turned off all rate limiting on my WAN router and used m1 = 25%, d = 30, m2=20% and even higher values (40%) and am still unable to get acceptable levels of quality. ;( That's with my qlanroot set for 1200k, which is about 20% less than what the downlink is. I've even tried qlanroot=1000k (33%less than downlink) and still quality suffers.
I've checked the queues and see that bulk file transfer is placed in a low queue and that VoIP is being placed into the qVOIPUp/Down queues as well.
Any other ideas? I'd love to get this working inside pfSense without having to resort to external scripts, etc.
Thanks again!
-
If possible can you try the values without qACK queue at all or if you cannot get rid of it, please try setting its d parameter two something twice thar of qVoIP.
I am very interested in your findings.
Thanks
-
Create an alias with all of your VOIP phones and servers in it. During the traffic shaping wizard, feed it to the red alias box on the VOIPStep. This will hopefully get the bulk data in the VOIP queues.
-
@eri--:
If possible can you try the values without qACK queue at all or if you cannot get rid of it, please try setting its d parameter two something twice thar of qVoIP.
I am very interested in your findings.
I removed the ACK queues and re-tested. Here's the current status report:
1.) speedtest w/o Traffic Shaping enabled = 1292 Down / 820 Up
2.) speedtest w/ qlanroot=1200, qwanroot=700: 1160 Down / 644 Up
3.) speedtest w/ qlanroot=1100, qwanroot=700: 1080 Down / 643 UpWith a high-speed download, glitching occurs even with the settings for 2 and 3, and with the ACK queues deleted. My assumption is that VoIP still isn't getting enough "headroom". I tested this theory by doing something extreme:
4.) speedtest w/ qlanroot = 500, qwanroot = 400: 489 Down / 376 Up
For setting #4, VoIP is perfect with saturated download - but my bandwidth is hampered for as long as shaping is enabled. This is the reason I wrote the Snort scripts, so that I can have perfect VoIP without having to sacrafice 50%+ bandwidth all the time.
Here's a different way of looking at it. Is it possible to place some "multiplier" on high priority traffic? Let's assume the VoIP takes 100k of bandwidth. Could the shaper multiply that bandwidth by a factor of 5, and use the 500k value for it's traffic shaping purposes? Or, if the VoIP takes 32k, then be able to give that a 15x multiplier for computational purposes?
I'm really looking to starve/penalize the bulk queues (actually all queues, for that matter) for the duration of the VoIP call. It seems to be the only way to get perfect VoIP in times of download saturation.
Let me know if you'd like me to try any other suggestions. Thanks!
-
Create an alias with all of your VOIP phones and servers in it. During the traffic shaping wizard, feed it to the red alias box on the VOIPStep. This will hopefully get the bulk data in the VOIP queues.
Thank you for the suggestion. I assume you want me to do this because you're not sure if the VoIP traffic is actually showing up in the VoIP queue. I've checked my previous configurations and VoIP falls in the right place. I've watched the queues page while making calls and see the traffic falling into the correct queue - so that's not the problem. That being said, I still tried this out:
The Vonage adapter is the only device going across the pfSense box and out to the Internet. I checked diagnostics/states to determine the IP address of the Vonage adapter (it's the only one using UDP 10000). According to your suggestion, I created a firewall alias for this address and re-ran the Traffic Shaper Wizard. I provided the alias to the red box in the VoIP section.
This provided no noticeable change from the previous posting. I see traffic in the VoIP queue when I make a call. Unfortunately, VoIP traffic still glitches.
I should probably define "glitches"… incoming audio (download) skips due to packet delay/loss. Without shaping at all, things are quite bad. With shaping enabled (and a sensible qlanroot), things are mostly intelligible, but there is still an unacceptable level of audio artifacting. When qlanroot is set for something extreme, the artifacting goes away and call quality is perfect... unfortunately at the permanent cost of bandwidth.
Any other suggestions? Thanks!
-
Well i forgot to tell you that you need to tweek the qlimit also.
The scheduler that is on pfSense 1.2 creates configuration differently form the one that i wrote so please try the suggestion below:I am assuming only one phone so from the discussion on ACK queue on the same category on this thread we have:
N = 1
D = 10ms
S = 150 bytes
m1 = 8 * S / D
than m1 = 8 * 150 / 10 = 120 Kbit/sd = (N+2) * D
d = (1 + 2) * 10 = 30ms
So your config should look like.
No ACK queue. For qVoIP{up,down} queue set m1 = 120Kb d=30 m2 = 75KbCan you please tweak the qlimit also of q{Wan,Lan}root and qVoIP{up,down} queue to something like 50 +-20% instead of 500.
Play with that since IT SURELY is why you're phone works fine when you lower bandwidth. So you will not need to do hard scripting. -
Please set even m1 = m2 on the q{Wan, Lan}root queue.
Seems that all this should do.
-
@eri--:
No ACK queue. For qVoIP{up,down} queue set m1 = 120Kb d=30 m2 = 75Kb
Easy enough. I assume that m1/d/m2 are located in the "realtime" Service Curve, not one of the others (upperlimit, link share)?
@eri--:
Can you please tweak the qlimit also of q{Wan,Lan}root and qVoIP{up,down} queue to something like 50 +-20% instead of 500.
Where can I make the change for qlimit? I did a quick search of the forums for "qlimit" and didn't see where I might be able to change that. It is not located in the GUI (that I can tell). I've hand edited /cf/conf/config.xml, but that file seems to be rebuilt by some other process. I looked at all of the files in /tmp, but none of them have a "qlimit" value either.
Which parent file can I edit to adjust the qlimit? When I know this, I can attempt these suggestions. Thanks again for your help.
-
@eri--:
Please set even m1 = m2 on the q{Wan, Lan}root queue.
After re-running the traffic shaper, there are no values for m1 or m2. All that are in this screen are:
Scheduler Type HFSC
Bandwidth = 800Kb (or 1200Kb for qwanRoot)
Priority = 0
name qlanRoot (or qwanRoot)
Scheduler Options: "This is a parent queue"(Also see attached .jpg for qwanRoot)
All other values are blank. Should I be populating m1 and m2? What value should I use?
-
Get a backup of your config after creating the configuration i told you.
You can get a backup from System->Backup or something similar.Edit that file and restore it back on the section where you got the backup.
This should allow you to make the changes made by hand to the config.xml be respected by pfSense.
And yes, i am talking about realtime specifications.
Try this and let me know.
Edit: Yeah you should fill them yourself. But for q{Wan, Lan}root this modifications should be done to the linkshare curve.
-
To explain even better:
1- calculations of realtime service curve should only be applied to qVoIP{up,down}
2- m1 = m2 should only be applied to q{Wan, Lan}root linkshare service curve. -
@eri--:
m1 = m2 should only be applied to q{Wan, Lan}root linkshare service curve.
What values should I place in here? Am I using the same values for the q{Wan, Lan}root "Bandwidth" section, ie: m1=m2=1200Kb for qwanRoot, and m1=m2=800Kb for qlanRoot - or is there some other value to be placed here?
-
To explain even better:
1- calculations of realtime service curve should only be applied to qVoIP{up,down}
2- m1 = m2 should only be applied to q{Wan, Lan}root linkshare service curve. They should be equal to the bandwidth specification on the queue and make d = 200.
3- qlimit to all of the queues or at least those mentioned here -
hidden any luck with this?!
-
According to this source http://www.dslreports.com/forum/remark,10277184
the Vonage adapter may need more bandwidth: at least m2 = 87.2 kb/s and m1 = 174.4 kb/s (for D=10 ms). -
Thanks for all of the suggestions. That seemed to help quite a bit. If I set my root values to 1000 or 1100Kb, then the VoIP has acceptable quality. Setting it at 1200Kb is still a little choppy.
Unfortunately, one of these settings has caused some adverse side effects:
1.) some VoIP call-setup doesn't work and I get fast busy
What's the ETA for a build with the "new" shaper? Any other suggestions?
(Edit: webpage issue removed - unreleated to traffic shaping).
-
1.) some embedded images in webpages don't show up (server timeout, etc.)
2.) some VoIP call-setup doesn't work and I get fast busyWell can you tell me the settings you used?!
For 1200Kb that's normal if you're on DSL since you don't get it all.
So it seems that qlimit does it for you. I would suggest to increase the qlimit for the q{Wan, Lan}root as i said by 20 or even 30% and of the queue where http traffic goes.
To fix the webpages get back the qACK on both WAN and LAN and set its realtime to 100ms with a m2 of 20% for each of them. This should get back your web pages.One downside of the shaper in 1.2 is that you cannot control the qlimit and tbr of the tocken bucket which would give you the perfect setup for your environment.
Try to increase the m1 and m2 with what dusan suggested or if you don't mind some more to get propper quality and bandwidth. You can even try to increase the latency to some more since it seems you have a higher latency on your line. You can be safe till 50ms that is VoIP upperlimit on average.This is to give you a better setup on current pfSense. I am working on finishing it and even waiting for the bounty people to complete thier commitement so i cannot give a ETA for that, sorry.
-
According to this source http://www.dslreports.com/forum/remark,10277184
the Vonage adapter may need more bandwidth: at least m2 = 87.2 kb/s and m1 = 174.4 kb/s (for D=10 ms).I'm currently using Vonage 90 codec. When you say "at least", would that mean that using values higher than M1=87.2Kb and M2=174.4Kb should reserve more bandwidth for VoIP? Like I've mentioned before, I would love it if I could slow <everything>down while I'm on a VoIP call… wasted bandwidth during the call is completely acceptable and is exactly what I want to do. If that is the case, could I just use M1 & M2 numbers of 512Kb? </everything>
-
I would suggest you to reread the thread where ACK for HFSC queue is being discussed, the first thread on the sticky tags.
Then get an idea of what it means to have that high bandwidth.
But the safer bet it experimenting different values. You can get some more complex with upperlimit too but just read that and ask what you do not understand.
-
@eri--:
Well can you tell me the settings you used?!
I need to take a step back. When I restored the settings after hand-editing the config.xml, the pfSense box reloaded and my snort scripts were in the startup director (explains the great call quality).
Now that I have that disabled, here's the current status:
no ACk queues
qlanRoot/qwanRoot = 1000/800 (with linkshare set for m1 1000, d 200, m2 1000)
qVOIPUp/qVOIPDown = 25% Bandwidth, prio 7, Realtime M1 240Kb, d 30, m2 240Kb
qlimits on all the queues = 50 (default, voip, p2p, etc.)Still choppy voice. I'll read some more about the HFSC & ACK Queues as you've suggested. I'm about ready to throw in the towel and add $100 to the bounty to get the new shaper out. ;) I'll let you know what I come up with.
-
According to this source http://www.dslreports.com/forum/remark,10277184
the Vonage adapter may need more bandwidth: at least m2 = 87.2 kb/s and m1 = 174.4 kb/s (for D=10 ms).I'm currently using Vonage 90 codec. When you say "at least", would that mean that using values higher than M1=87.2Kb and M2=174.4Kb should reserve more bandwidth for VoIP? Like I've mentioned before, I would love it if I could slow <everything>down while I'm on a VoIP call… wasted bandwidth during the call is completely acceptable and is exactly what I want to do. If that is the case, could I just use M1 & M2 numbers of 512Kb?</everything>
In the realtime curve, every bandwidth requirement like m1 and m2 imply a delay requirement and to set it to higher value simply means to require lower delay. Actual delay in the worst case, however, cannot be guaranted to be shorter than link delay, which is the time needed to complete previous transmission request. For 1200 kb/s link with MTU = 1.5 kB = 12 kb, the link delay is not larger than 12/1200 = 0.01 s = 10 ms. By requiring lower delay (e.g. 1 ms), you won't get an absolute guarantee but you'll have higher chance that the actual delay will be short.
Requiring high real-time bandwidth does not mean wasting the link's bandwidth. Regardless of what m1 and m2 you specify, your codec always runs at constant bitrate of 87.2 kb/s (that's why is is named Vonage 90). The rest of bandwidth is allocated to other traffics.
-
One simple question:
Did you remove realtime specification from the other queues?! -
According to this source http://forums.whirlpool.net.au/forum-replies-archive.cfm/331189.html
there maybe also two problems.- Bad queue length limit. Beside previously mentioned 'qlimit' there may be other (hardware) queue length limits that may need tweak:
-
in pfsense machine's WAN NIC. Optimum may be 50-70.
-
in switch, if VoIP adapter is connected to pfsense machine through a switch. The best solution is to reconnect the adapter to pfsense machine directly.
- Bad packet fragmentation. Beside tweaking MTU, I don't know if there are other means to control fragmentation. For slow uplink (< 1200 kb/s) fragmentation may need tweak by reducing MTU in pfsense's WAN NIC. The default MTU (not counting Ethernet overhead) is 1500 bytes, which is optimal for normal data (web etc.) traffic. For the Vonage 90 the optimal is 200 bytes. One may need to choose a compromise between them.
-
You don't need to do 2) with altq since there is a tocken bucket which may compensate the MTU problems.
While to me 2) is somewhat 'ridiculously strange' ALTQ provides the means for it and according to my knowledge that is a problem with modern drivers which try to burst aggressively.
The shaper in pfSense 1.2 does not allow for this so you cannot tweak it.For 1) i cannot say other than drop the adapter and use some form of software VoIP client. Getting down to such details is somewhat paranoid behaviour so please do not scare people in such ways :). Though, it is true that enterprise class hardware and home users ones do really have 'big' differences.
-
Ok here is another one to try, slightly different setup!
qlanRoot: linkshare 1024Kb/s (this is parameter m2 no bandwidth set, if you can set m1 = 0 and d to your link latency might help).
qwanRoot: linkshare 800Kb/s (this is parameter m2 no bandwidth setif you can set m1 =0 and d to your link latency might help).
qVoIP{up, down}: linkshare( m1 = 1.3Kb, d = 5, m2 = 120Kb ) realtime( m1 = 1.3Kb, d = 5, m2 = 120Kb )
qACK{up, down}: linkshare(m1=500b, d = 100, m2 = 25%) 'it is b == bits/s
on the other queues do whatever you want just don't set realtime paramter.Can you please test this, i am very interested in your results and this might save you 100$ :).
-
Anybody tried this?!
-
Im watching this thread with great interest. If I get time Ill try it when I get home.
Im a voip system tester so have a few lines coming in here.
-
@eri--:
qlanRoot: linkshare 1024Kb/s (this is parameter m2 no bandwidth set, if you can set m1 = 0 and d to your link latency might help).
qwanRoot: linkshare 800Kb/s (this is parameter m2 no bandwidth setif you can set m1 =0 and d to your link latency might help).
qVoIP{up, down}: linkshare( m1 = 1.3Kb, d = 5, m2 = 120Kb ) realtime( m1 = 1.3Kb, d = 5, m2 = 120Kb )
qACK{up, down}: linkshare(m1=500b, d = 100, m2 = 25%) 'it is b == bits/s
on the other queues do whatever you want just don't set realtime paramter.Can you please test this, i am very interested in your results and this might save you 100$ :).
Sorry, I've been busy for the last few days. I'll try and get to this sometime today. I'll post back with the results. Thanks!
-
@eri--:
qlanRoot: linkshare 1024Kb/s (this is parameter m2 no bandwidth set, if you can set m1 = 0 and d to your link latency might help).
qwanRoot: linkshare 800Kb/s (this is parameter m2 no bandwidth setif you can set m1 =0 and d to your link latency might help).
qVoIP{up, down}: linkshare( m1 = 1.3Kb, d = 5, m2 = 120Kb ) realtime( m1 = 1.3Kb, d = 5, m2 = 120Kb )
qACK{up, down}: linkshare(m1=500b, d = 100, m2 = 25%) 'it is b == bits/s
on the other queues do whatever you want just don't set realtime paramter.qlanRoot/qwanRoot results
First off, if I set Bandwidth = (blank) and Linkshare = 1024Kb/800Kb, filter errors occur:There were error(s) loading the rules: /tmp/rules.debug:13: syntax error/tmp/rules.debug:16: queue qwanRoot has no parent /tmp/rules.debug:16: errors in queue definition /tmp/rules.debug:18: queue qwandef has no parent /tmp/rules.debug:18: errors in queue definition /tmp/rules.debug:20: queue qwanacks has no parent /tmp/rules.debug:20: errors in queue definition /tmp/rules.debug:22: queue qVOIPUp has no parent /tmp/rules.debug:22: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [13]: altq on xl0 hfsc bandwidth queue { qwanRoot }…
If I set Bandwidth = 0 and LInkshare = 1024Kb/800Kb, then I lock myself completely out of the pfSense box. I had to un-rack it, reset to factory defaults, and re-run through the configuration wizard. ???
qVOIPUp/Down results
Realtime m1 = 2Kb, D = 5, and m2 = 120Kb I get the following error:There were error(s) loading the rules: pfctl: m1 must be zero for convex curve: qVOIPUp/tmp/rules.debug:23: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ m1 must be zero for convex curve]: …
If I remove realtime m1 and d, and leave 120Kb for m2 then the filter loads properly.
If realtime m1 = 0Kb, d=50, and m2 = 1024Kb then I receive the following error:
There were error(s) loading the rules: pfctl: linkshare sc exceeds parent's sc/tmp/rules.debug:19: errors in queue definition pfctl: linkshare sc exceeds parent's sc /tmp/rules.debug:21: errors in queue definition pfctl: linkshare sc exceeds parent's sc /tmp/rules.debug:23: errors in queue definition pfctl: linkshare sc exceeds parent's sc /tmp/rules.debug:25: errors in queue definition pfctl: linkshare sc exceeds parent's sc /tmp/rules.debug:27: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ linkshare sc exceeds parent's sc /tmp/rules.debug]: …
Next, if Realtime m2 = 120Kb, and Linkshare m1 = 2Kb, d=5, m2=120Kb then:
There were error(s) loading the rules: pfctl: m1 must be zero for convex curve: qVOIPUp/tmp/rules.debug:23: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ m1 must be zero for convex curve]: …
qwan/lan acks results
If I use Linkshare m1=2Kb, d=5, m2=120Kb I receive:There were error(s) loading the rules: pfctl: m1 must be zero for convex curve: qwanacks/tmp/rules.debug:20: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ m1 must be zero for convex curve]: …
If I use Realtime m1=2Kb, d=5, m2=120Kb I receive:
There were error(s) loading the rules: pfctl: m1 must be zero for convex curve: qwanacks/tmp/rules.debug:20: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ m1 must be zero for convex curve]: …
It looks like Realtime can only have m2 value, nothing for m1/d
Further, if I use a combination of Realtime m2=120Kb, and then Linkshare m1=2Kb, d=5, m2=120Kb I receive this error:
There were error(s) loading the rules: pfctl: m1 must be zero for convex curve: qwanacks/tmp/rules.debug:20: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [ m1 must be zero for convex curve]: …
By the way, 1.2Kb is not allowed, nor are "bits" an option in the GUI (thus, the reason why I used 2Kb instead).
The snort option (http://www.xmission.com/~hidden/aatqos/) looks a lot easier than this so far. ;D
It goes without saying that due to all these errors, I am unable to "test" anything. Any other ideas? Thanks again for all your help.
-
Those are all pfctl problems i will post a link to a pfctl which doesn't have these problems so you can test it again, if you want to do that.
From my testing this configuration is the way to go but seems PF guys have misunderstood some things about HFSC.
Btw, thanks for your help on this. I just want to test it as much as possible so on 1.3 you just need the wizard and get done.
qlanRoot/qwanRoot results
First off, if I set Bandwidth = (blank) and Linkshare = 1024Kb/800Kb, filter errors occur:There were error(s) loading the rules: /tmp/rules.debug:13: syntax error/tmp/rules.debug:16: queue qwanRoot has no parent /tmp/rules.debug:16: errors in queue definition /tmp/rules.debug:18: queue qwandef has no parent /tmp/rules.debug:18: errors in queue definition /tmp/rules.debug:20: queue qwanacks has no parent /tmp/rules.debug:20: errors in queue definition /tmp/rules.debug:22: queue qVOIPUp has no parent /tmp/rules.debug:22: errors in queue definition pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [13]: altq on xl0 hfsc bandwidth queue { qwanRoot }…
If I set Bandwidth = 0 and LInkshare = 1024Kb/800Kb, then I lock myself completely out of the pfSense box. I had to un-rack it, reset to factory defaults, and re-run through the configuration wizard. Huh
this might be a shaper rule generation error.
-
@eri--:
Those are all pfctl problems i will post a link to a pfctl which doesn't have these problems so you can test it again, if you want to do that.
Btw, thanks for your help on this. I just want to test it as much as possible so on 1.3 you just need the wizard and get done.Glad to help. I'll be able to do some testing today & Wednesday, and then won't be able to test again until Feb 5. I'll watch for the link to a pfctl. Thanks.
-
Grab pfctl command, from here that allows you to set m1 smaller than m2. So you can try my posted config.
For the q{Lan, Wan}root use what didn't gave you errors.The procedure is simple:
1- on your pfSense machine login with ssh
2- select option 8
3- copy /sbin/pfctl to /sbin/pfctl_old
4- scp the attached pfctl to /sbin
5- retry the previous suggested config and report.Thanks.
-
I tried so many different ways with the original pfctl - no dice. the calls were consistently bad when i saturated the pipe. as soon as i replaced pfctl with the above version, everything worked great! No more call quality issues! I just made the change today. I'll be back with more details of my settings once i've confirmed it works for a few days. i have 5 phones behind pfsense connected to a public Asterisk server on the other side of the US (DC to LA). My connection is a T1 over the Internet (not a private line).
-
The afforementioned pfctl is no longer there, does someone have a copy to share?
-
Be patient soon a alpha version will be released that has this fix.
If you feel like an alpha tester, i suppose you are since are willing to copy extraneous binaries to your firewall, than be ready for it. -
@ermal:
Be patient soon a alpha version will be released that has this fix.
If you feel like an alpha tester, i suppose you are since are willing to copy extraneous binaries to your firewall, than be ready for it.Thanks!
-
Just got a report back from the users that it is not perfect, but a lot better! These problems may not be relating to shaping though. I'm thinking my PSTN gateway provider may not be perfect.