Another question about choppy audio on Zoom, Teams & Slack

charry2014 · Nov 13, 2020, 4:47 PM

My home network is hugely over dimensioned for two adults - but still we get very poor quality audio calls on wired and wifi connections to Zoom, Teams & Slack. All are poor to the point of being almost unusable.

The firewall is PFSense 2.4.5-RELEASE-p1 on a 4 core Intel Celeron N3150. My internet connection is a 200/100MBit fibre from Deutsche Glasfaser. The main switch is a gigabit Netgear M4100, all wiring in our house is CAT.7. The Wifi network is 3x Ubiquiti nanoHD APs with the controller version 6.0.22 running in Docker on a Linux machine. This should be able to handle 2x simultaneous calls - most often over Teams and Slack.

What have I tried so far:

Run through the PFSense Traffic Shaper Wizard and added configured some basic settings to prioritise and lower certain traffic types
Added PFSense FQ_CODEL limiters on WAN and LAN as described in this video from Netgate
Made sure that jumbo frames are turned off everywhere
Tested on wired and wireless connections
Run speed tests with speedtest.net and dslreports.com/speedtest which show mixed results - see below
Running ping tests with PingPlotter against teams.microsoft.com - these show frequent packet loss as below.
Run test calls to Teams' echo caller and listened for drop outs - which correspond to what PingPlotter shows
Updated everything in the network and rebooted everything

The dslreports.com speed test does reveal sometimes that the BufferBloat and Speed scores are poor even though the throughput in MBit/s usually gets very close to the 200/100 MBit/s I expect.

PingPlotter shows frequent dropouts to teams.microsoft.com, when the audio goes crunchy -
🔒 Log in to view
Unhelpfully none of the Slack endpoints respond to pings.

So given all this I am at a loss what to do next - it starts to look like the problem rests with the internet provider, but the link is mostly really fast. What might be going wrong here?

Rico · Nov 13, 2020, 5:47 PM

I have hundred Users doing Zoom and Teams stuff simultaneously all day long, so there can't be any problem with pfSense in general using those services.
What NICs are you using?
Did you try with vanilla pfSense settings to rule out any config or package problem?
Can you skip pfSense and connect your PC directly to the ISP device for testing?
Do you also see loss from pfSense directly, ping teams.microsoft.com from the pfSense console (SSH)?

-Rico

Rico · Nov 13, 2020, 6:04 PM

My home pfSense after ~15min (1121 packets):

64 bytes from 52.113.194.132: icmp_seq=1118 ttl=119 time=8.160 ms
64 bytes from 52.113.194.132: icmp_seq=1119 ttl=119 time=7.659 ms
64 bytes from 52.113.194.132: icmp_seq=1120 ttl=119 time=7.765 ms
^C
--- s-0005.s-msedge.net ping statistics ---
1121 packets transmitted, 1121 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.543/8.053/11.745/0.228 ms
[2.4.5-RELEASE][admin@pfsense.xxx.local]/root:

-Rico

charry2014 · Nov 13, 2020, 9:16 PM

Thank you for your answers. Everything ran fine for quite a while with a relatively vanilla pfSense and I only started fiddling with it when the problems started - but a sensible next step seems to be to patch a few ports directly through to the provider's network and test the connection there. I asked my neighbours who are signed up to the same provider and none have any problems.

One thing I never found a straight answer to - do voice calls over Teams, Slack and Zoom use the network in the same way that VoIP calls do? I am never sure that when articles on the internet are written about VoIP or a firewall setting applies to VoIP if that will also be relevant to this issue.

SteveITS · Nov 13, 2020, 10:39 PM

In the traffic shaper is the bandwidth setting high enough for the video?
https://forum.netgate.com/topic/158080/why-does-bandwidth-setting-affect-call-quality

charry2014 · Nov 16, 2020, 7:58 AM

Thank you for the suggestion - I have set the queue lengths in the traffic limiters to 2000 and run through these videos on traffic shaping with the pfSense wizard and scheduling with FQ_CODEL as in the video from Netgate. These two are configured correctly. I have tried dropping my the speeds for in & out on my internet connection to something that resembles what the speed tests report I get. Now I will let this alone for a while to see how the calls run today.

A specific question about the floating firewall rules and the coexistence of the limiter rules and those created by the wizard - currently it seems all traffic is being hit by the WAN rule for the limiter and nothing hits the other rules. All the rest show 0/0B.

The WAN rule for the limiter is at the top:
🔒 Log in to view
Should I move this down to the bottom?

charry2014 · Nov 16, 2020, 1:52 PM

Eventually this whole problem made no sense any more and in desperation I rebooted the pfSense - problem solved. No packet loss, calls working fine.

In the good old days when things crashed and broke all the time rebooting stuff was the first thing to try, I have got out of practice at that.

If someone can answer my question about the firewall rules above I would appreciate it, but thank you for your help.

SteveITS · Nov 16, 2020, 3:40 PM

The States on the floating rules have been way off at times (https://forum.netgate.com/post/942173). What does status/queues show for the VoIP queue when you're on a call? I would trust that page far more than the States column.

charry2014 · Nov 17, 2020, 1:51 PM

Interesting - Status / Queues shows the following:
🔒 Log in to view

Also weird the P2P queue is so busy - I can't think of anything that specifically uses that - but none of the other queues ever register traffic.

SteveITS · Nov 17, 2020, 3:17 PM

qP2P is presumably low priority...
I would read through https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html, in particular the difference with Quick enabled/disabled.

It's possible your P2P rules are picking up some traffic by port number but not all of it. Is it set as your default queue or something? I'm unclear what the top two rules are for, would they not match all upload traffic? In what queue do those put traffic?

charry2014 · Nov 19, 2020, 10:52 AM

Having rebooted pfSense the queues seem to be working fine now, also the call quality with Teams, Slack and Zoom is acceptable. There was a lot of traffic on qP2P as the catch-all option was set. These things are now working fine.

A problem with qVoIP remains - during calls with Slack, Teams and Zoom no traffic is routed through this queue. There is no rule in the firewall by the traffic shaper wizard:

🔒 Log in to view

Any idea why? Do Slack, Teams and Zoom even qualify as 'VoIP' services?

SteveITS · Nov 19, 2020, 4:38 PM

In the shaping wizard there was an option for VoIP and has one enter the remote IPs. Otherwise there's not a great way for pfSense to know what is VoIP traffic. And since you don't know what IPs all of those use it becomes difficult to maintain.

One option might be to prioritize all UDP traffic from your device using those services, but there is a caveat noted in the docs, that the shaper works on outgoing traffic and on the WAN (upload) that happens after NAT. So you can't use your private IPs in the rule that applies the outgoing shaping. What you can do is tag the packets from those IPs, and use that tag. https://docs.netgate.com/pfsense/en/latest/trafficshaper/advanced.html#shaper-rule-matching-tips

rule with source of your PC IP:
🔒 Log in to view

rule with source and dest of Any that only applies to the tag, and assigns the queue:
🔒 Log in to view