Hardware problem or ISP problem - Drops in audio on VoIP
-
I have a plaguing Asterisk VoIP call quality problem. Our audio outbound via the SIP trunk is very choppy/drops in audio. Inbound is generally good.
I don't believe it is the Traffic Shaper configuration as there are NO drops indicated on any queue and traffic appears in all the correct queues during calls.
I am using a 2.66Ghz Intel processor, with a standard PCI bus.
One onboard ethernet (FXP), 2 RL based cards and one DC based card.
I have a 4 port Intel Server card I can access to put into the pfSense router if that would make a difference. The WAN interface (RL card) shows 3 errors on the interface in 76 days.Our main Internet connection is a 4M/4M fixed point WiMAX link. Opt1 goes to a 100Mbit DMZ with 2 servers on it. Opt2 is connected to a Netgear Router for an adsl connection that we have routed via a pfSense rule ALL http and https browsing traffic out that is not going to one of the DMZ servers. The LAN (fxp) connection is connected to a gigabit port on the PoE switch.
Looking at the RRD graph for Quality, I usually see up to 1.5% dropped packets to the ISP gateway in a given 4 hour period. I received a second IP address from the ISP and have put in a small switch between the pfSense router and the Radio for the WiMax connection for troubleshooting. From that connection, I have done a very, very short test and saw no problem with dropped packets on a Skype call displaying technical info. Skype calls usually show .3% packet loss on sending. (I probably should do some more tests though.)
RRD Graphs for CPU are always under 8% and usually 3% or less. RRD Graphs for total traffic usually shows about 400k incoming and around 100k outgoing which makes it appear as we are not saturating the bandwidth.
I suspect it is the WiMax ISP, however after my little test outside the router, they are asking me to put in a Cisco router as they think they have done enough to fix the connection on their end.
Questions:
1. Would changing out the NICs be beneficial to this situation when I am not showing any loss of packets?
2. Should I be looking somewhere else for problems?
3. Are there other ways to determine if the Router or ISP's Radio or the ISP is dropping the packets?
4. Should I just suck it up and buy an expensive multi-wan Cisco router? (I hope not)
5. Is accessing the DMZ servers not going through the Queues and causing problems? -
You already have a switch between the radio and the pfSense box. I'd debug this using Wireshark and a switch that can port mirror - mirror the port that you have the radio connected to another port, then connect a machine running Wireshark to the mirror port to analyse the VoIP traffic. As this is SIP, Wireshark will make this easy for you, as it has extensive built in support for RTP.
If you don't have a suitable switch, the inexpensive HP Procurve 1800-8G will do the job. Though I don't own one myself, there have been complimentary posts about it in the Hardware forum recently, also it's inexpensive, 8 port Gigabit and it supports VLANs (so you can use it as a 'breakout box' for a pfSense machine with a quality VLAN enabled Gigabit NIC, such as an Intel server NIC, rather than using multi-port NICS using less reliable controllers). These little Procurve switches are so cheap (they're available in the UK for less than £70) that it's worth having one as part of your debugging kit if you have anything other than a trivial network to support.
If Wireshark shows no packet loss on the outbound audio, tell the ISP to fix their system. If it shows packet loss, the problem is somewhere in your pfSense machine. This is a definitive test, assuming that the switch you're using has a good port mirroring implementation. 4Mbps in each direction shouldn't stretch the mirroring capabilities in any decent switch.
If you don't have a machine to hand that you can run Wireshark on, there are live CDs that include Wireshark that you can boot on a laptop.
I would change the NICs and cables anyway. If you've got a spare four port Intel Server NIC, I'd use it. rl(4) network interfaces really aren't that good - they're inexpensive consumer NICs that turn out to be responsible for quite a few pfSense problems. dc(4) is an older type of NIC - you may well be better off with a modern one.
-
If you cannot mirror a port, you could also use on of these old hubs lying around :D
( I always keep one or two of these around just in case… ;) ) -
I am fairly unfamiliar to troubleshooting VoIP, but am familiar with WireShark. Forgot I could do that on the WAN interface of the router since it is Ethernet. I have a 1.2GHz Laptop with 2 Ethernet ports in bridge mode with WireShark installed that I have used in the past (Dell C series in Docking Station). Thanks for the reminder that I could use it here. I do have a 10Mbit Hub I can use if I need too also, however I prefer to keep it in Full Duplex cabability on the router.
Sometimes it just takes another brain to approach it from a different angle, thanks for the advice. I will try it on Monday and Tuesday.I'll let you know how it goes.
Kent
(PS. I love Procurve Switches. I bought and installed a 1200+node Procurve network at my former work location. I have for 15+ years been the LAN guy. Just recently started doing WAN stuff at a smaller office with same org.)