Optimize MS Teams calls
-
Hi, I've been looking into some performance problems we have been having since a long time. They only got worse as the company grew overtime. My main focus is to solve MS Teams issues: video / audio calls often drop or the quality is (very) bad.
My setup
- 300/30 mb
- 20-ish users, of which at most 6 are in in a Teams call at the same time
- heavy web browsing and other office tasks, but no other specific traffic
- system: Netgate 7100, 22.05-RELEASE (amd64)
I've read through many topics on this issue, and did the following:
- Set firewall optimization to conservative
- Set up FQ_Codel according to these instructions: https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/814 (see screenshots below for detailed config)
My problem
I test by running the MS Teams assessment tool: https://www.microsoft.com/en-us/download/confirmation.aspx?id=103017 :
NetworkAssessmentTool.exe /qualitycheckIt starts fine, but then I start a heavy download on another machine, and the latency increases significantly. See result below.
TIMESTAMP is in UTC. LOSS RATE is in percentage, out of 100. LATENCY and JITTER are in milliseconds, and are calculated as averages in ~5-second windows. PROTOCOL displays whether UDP, TCP (PseudoTLS/FullTLS), or HTTPS protocol was used to allocate with the relay server. Note that for PROTOCOL, UDP protocol is attempted first to connect to the relay, by default. LOCAL ADDRESS is the local client IP and port that media is flowing from. REMOTE ADDRESS is the peer (relay server) destination IP and port that media is flowing to. IS PROXIED PATH shows whether a proxy server is used to connect to the relay, only applies to TCP/HTTPS connections LAST KNOWN REFLEXIVE IP shows what your latest public (NAT translated) IP and port is that the relay sees during media flow. [If LOSS RATE is 100%, the output lines here will be in red] Quality check source port range: 50000 - 50019 Call Quality Metrics: 2023-02-04 10:14:35 Loss Rate: 0 Latency: 77,2 Jitter: 153,38 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:14:44 Loss Rate: 0,44 Latency: 110,58 Jitter: 209,03 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:14:52 Loss Rate: 0 Latency: 53,42 Jitter: 157,41 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:00 Loss Rate: 0 Latency: 31,03 Jitter: 209,72 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:09 Loss Rate: 0 Latency: 30,03 Jitter: 150 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:17 Loss Rate: 0,44 Latency: 68,16 Jitter: 210 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:26 Loss Rate: 0 Latency: 32,25 Jitter: 72,82 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:34 Loss Rate: 0 Latency: 22,42 Jitter: 29,47 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:42 Loss Rate: 0 Latency: 27,72 Jitter: 73,37 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:51 Loss Rate: 0 Latency: 245,06 Jitter: 599,83 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:15:59 Loss Rate: 0,44 Latency: 1049,89 Jitter: 1711,37 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:08 Loss Rate: 0 Latency: 98,88 Jitter: 234,32 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:16 Loss Rate: 0,44 Latency: 73,98 Jitter: 91,28 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:24 Loss Rate: 0 Latency: 23,48 Jitter: 24 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:33 Loss Rate: 0 Latency: 23,45 Jitter: 31,09 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:41 Loss Rate: 0 Latency: 22,68 Jitter: 17 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:50 Loss Rate: 0 Latency: 22,41 Jitter: 16,87 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:16:58 Loss Rate: 0 Latency: 22,76 Jitter: 31,61 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:06 Loss Rate: 0 Latency: 22,55 Jitter: 19,95 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:15 Loss Rate: 0 Latency: 23,19 Jitter: 28,84 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:23 Loss Rate: 0 Latency: 22,52 Jitter: 19,51 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:31 Loss Rate: 0 Latency: 22,68 Jitter: 21,41 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:40 Loss Rate: 0 Latency: 22,76 Jitter: 16,72 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:48 Loss Rate: 0 Latency: 22,59 Jitter: 21 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:17:57 Loss Rate: 0 Latency: 22,08 Jitter: 19,92 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:05 Loss Rate: 0 Latency: 22,25 Jitter: 18 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:13 Loss Rate: 0 Latency: 21,48 Jitter: 23 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:22 Loss Rate: 0 Latency: 21,12 Jitter: 16,13 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:30 Loss Rate: 0 Latency: 21,04 Jitter: 18,63 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:38 Loss Rate: 0 Latency: 21,1 Jitter: 20,39 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:47 Loss Rate: 0 Latency: 21,06 Jitter: 21 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:18:55 Loss Rate: 0 Latency: 20,84 Jitter: 21,17 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:19:04 Loss Rate: 0 Latency: 20,84 Jitter: 17 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:19:12 Loss Rate: 0 Latency: 21,02 Jitter: 20 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:19:20 Loss Rate: 0 Latency: 20,83 Jitter: 19 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926 2023-02-04 10:19:29 Loss Rate: 0 Latency: 20,75 Jitter: 18,35 Protocol: UDP Local IP: 10.70.9.2:50009 Remote IP: 52.114.242.222:3478 Is Proxied Path: False Last Known Reflexive IP: xxx.xxx.xxx.xxx:39926MS recommended latency is below 60 if I recall correctly. The peaks of 250 and 1049 happened during the start of the download of a big file. If there is no other traffic (near the end of the test), the latency is fine. Bufferbloat test is near perfect ever since FQ_Codel was enabled:
But I don't care about getting a good score on Bufferbloat.
My Question
I would like to have my Teams calls working without issues :). Any ideas?
Screenshots of my FQ_Codel config
-
@mkcharlie right away I see snort enabled. I don’t know what your rulesets are but any package that requires that packets need to be scanned puts a burden on the processor.
Turn off snort and test.
Review the rules you have enabled in snort. You most likely won’t need half of them. -
@michmoor thanks! Now my forum signature dates from the old days, and when this new forum was created it could not be changed anymore. Perhaps that’s been fixed in the meantime and I should probably change it :).
Anyway I moved to suricata, so your comment is still valid. Still, I optimised the rules quite heavily and I would hope I can get a simple MS Teams call to work on the hardware I have, even with Suricata enabled. Note that CPU and RAM never break a sweat.
Up to date list of packages: Acme
Avahi
aws-wizard
Cron
freeradius3
ipsec-profile-wizard
ntopng
openvpn-client-export
pfBlockerNG-devel
suricata
Telegraf
WireGuard -
@mkcharlie YMMV but I can explain how I did it. In my case it was a few IPs and I didn't know the MS Teams servers so I did it gave the PCs static IPs, and matched on all outbound UDP. Not perfect but it covers VoIP and other audio also.
Traffic Shaper wizard using PRIQ (which simply forwards by priority). Select VoIP even if you have to put in a bogus server IP address (I'm not looking at the wizard right now) so it creates qVoIP.
Note there are a few bugs in the wizard:
- https://forum.netgate.com/topic/177040/wizard-causes-bug-on-voip-sip/1
- https://forum.netgate.com/topic/166621/priority-of-qotherslow-higher-than-default
- I've also seen posts to ensure all units are the same, i.e. Mbps, and a relatively even number, for instance not 1.3432 Mbps. The wizard tends to mix units and percentages.
Make an alias VoIP_Devices with your LAN computer IPs.
Floating rule: LAN, match, quick, UDP. Source=VoIP_Devices. Description="tag VoIP packets." Tag (under Advanced)="VoIPQ"
Floating rule: any, match, quick, UDP. Source=any. Destination=any. Description="VoIPQ by tag." Tagged="VoIPQ"
This method is from https://docs.netgate.com/pfsense/en/latest/trafficshaper/advanced.html#shaper-rule-matching-tips: "To match by a private address source outbound in WAN floating rules, first tag the traffic as it passes in on a local interface. For example, match inbound on LAN and use the advanced Tag field to set a value, and then use the Tagged field on the WAN-side floating rule to match the same connection as it exits the firewall. Alternately, queue the traffic as it enters the LAN with a pass rule instead of when it exits a WAN."
Floating rule: any, match, not quick, UDP, source=any, Destination=VoIP_Devices, Description="Connections to VoIP devices"
You should see the traffic in qVoIP in Status/Queues afterwards.
re: sig, I think you need enough likes/vote-ups to edit your sig, or something along those lines.
-
@steveits thanks!
First tests seem to have a positive result indeed, couple of questions:- Should / can I keep the Codel setup I did before, or do you suggest I remove that and only keep the PRIQ shaper I created based on your post?
- What should the exact order be of the floating rules? I now have your 3 rules first, the one created by the wizard fourth, and the Codel one last (see screenshot below).
- Theoretically I have 300/30, for Codel I set it to 250/25. For the new shaper I only used 250/25. Does that seem ok?
- I now match all devices in the officewired/officewireless VLANs. Theoretically all of these machines can use Teams. I suppose they also have other UDP traffic though. Do you see issues by making such a wide match?
- Are other rules created automatically by the wizard? I did not really understand the output. It seem to have done a lot.
-
@mkcharlie I should have mentioned, I have not really used CoDel. I had tried CBQ which functioned but also had some wizard errors and it was difficult to get the bandwidth working as expected, due partly to some wizard bugs. PRIQ is a basic "process higher level packets first" algorithm but has not been an issue. Perhaps it would if there were 500 PCs sharing a small upload pipe.
Floating rule order is different, last match wins unless quick is used. However tagging the packets is not an allow/deny scenario so it kind of doesn't matter.
For bandwidth generally one wants it below the max. So if you get 300/30 on a speed test (without other traffic) you could use 295/29 or something.
Download throttling of course is a bit out of your control since your router sees packets after they have come down the wire.
I figured matching UDP would get Zoom and other stuff too but that was OK. I haven't noticed any issues.
The wizard does generate rules based on the options selected. You can rerun the wizard (and it will recreate the old rules too). Usually I pick a low priority option like SMTP to create the queue even if I disable the rule, and same with others.
-
@steveits
Thanks! I did some more testing and the CoDel rule seems to work fine. Bufferbloat still gives nice scores.So I'll keep it like this and see next week for some real-world tests. Thanks for the fix!