VoIP with 3rd Party Provider
-
Hi everyone,
I have been using pfSense for years and absolutely love it! Recently my company decided to switch to a VoIP phone system. We are using a third party company called Fonality for our VoIP services. The system / service works well for the most part. On occasion however we are experiencing dropped calls at random intervals. I can test and recreate the issue but it seems to only happen about a third of the time when I test and the calls can last well over an hour before they drop.
We have about 20 phones in our main office and another 10 in a satellite office. I have done 2 things from the pfSense side to try and combat this. The first is to enable traffic shaping to prioritize VoIP traffic. The second is to set the Firewall Optimization Settings to Conservative. These two changes still have not seemed to totally resolve the issue.
After talking to the Fonality Tech team one thing they noticed in the logs that is a bit concerning is that the phones are seem to go to an unreachable state and then go back to reachable after a short time. See the log example below:
[Jul 31 00:16:39] NOTICE[30528] chan_sip.c: Peer '0004********' is now UNREACHABLE! Last qualify: 89
[Jul 31 00:16:43] NOTICE[30528] chan_sip.c: Peer '0008********' is now UNREACHABLE! Last qualify: 137
[Jul 31 00:16:43] NOTICE[30528] chan_sip.c: Peer '0008********' is now UNREACHABLE! Last qualify: 145
[Jul 31 00:16:43] NOTICE[30528] chan_sip.c: Peer '0008********' is now UNREACHABLE! Last qualify: 177
[Jul 31 00:16:54] NOTICE[30528] chan_sip.c: Peer '0008********' is now Reachable. (1174ms / 2000ms)
[Jul 31 00:16:54] NOTICE[30528] chan_sip.c: Peer '0008********' is now Reachable. (1102ms / 2000ms)
[Jul 31 00:16:54] NOTICE[30528] chan_sip.c: Peer '0008********' is now Reachable. (1177ms / 2000ms)
[Jul 31 00:16:58] NOTICE[30528] chan_sip.c: Peer '0004********' is now Reachable. (93ms / 2000ms)
[Jul 31 06:55:04] NOTICE[30528] chan_sip.c: Peer '0004********' is now UNREACHABLE! Last qualify: 29
[Jul 31 07:25:16] NOTICE[30528] chan_sip.c: Peer '0004********' is now Reachable. (31ms / 2000ms)Having zero experience with VoIP before this process started, I have been reading as much info as I can find and trying to compare it to my specific situation. My concerns lie with securing my network while making sure that our VoIP solution works as well as possible. Is there anything else from a pfSense side of things I need to try and configure?
Thanks for any / all suggestions!
Ben -
Take a close look into your RRD graphs for "quality" and see if you have internet drops or high latency during those periods.
-
As kejianshi said, review the quality of the RRD during that time is one check. The fact that you are connecting and operating normally except for periods of dropped calls etc indicates that the account itself and registration is setup. The questions I have would be 1) the phone system and service is both fonality? 2) what types of phone / models are you using? 3) what is your line speed up/down 4) is the voip system and phone in a separate subnet from computers/servers? 5) more problems occur during inbound or outbound calls? Those are some questions I have for now.
-
Hey guys,
Thanks for the quick replies. I do see some spikes in our latency (see attached image) but I can't match them up to dropped calls currently. Will definitely try to find a pattern of this going forward.
As far as the questions:
1. Yes both phones and service are provided by Fonality
2. We have 3 different model phones in use. They are the Polycom IP 331, the Polycom IP 5000, and the Astra 9480i CT.
3. 35MB up / 150MB down
4. No, our phones and computers are all on the same subnet
5. The calls that I am aware of dropping are all currently outboundAfter I originally posted this message, I read about VoipSpear.com which tests packet loss and latency to our network from 3 different servers and gives us a MOS score. It's been running for about an hour and so far there has been zero packet loss reported from the 3 servers and we have a MOS score of 4.3 from all servers, in case thats any additional help.
Thanks
Ben
-
All those little red dots running across the bottom of your chart are complete packet loss. Basically internet is dead at those moments.
Its happening alot on your network. (I'm pretty sure)I'm used to seeing alot of latency corresponding to those on my network when it happens.
Also, there are two ends to the connection. Even when yours are up, theirs can be down and they are unlikely to tell you if their network is POS or over subscribed.
-
And anything I see over 3-5 for VoIP is bad. We have several VoIP implementations, and those do not look good. We are talking about business grade services here, not home. Our clients avg ~500 per calls per day for 15-30 user environemnt. For less, about 200 calls, so we may notice more than others. VoIP service is the first service that you will notice affected by quality issues. 2 things, who's the ISP, contact them to resolve, but I would first try a different hardware for pfsense and ensure it's not hardware like NIC, etc. Make sure you use quality hardware for pfSense and Intel based NICs works best.
This is what you should see, stable and hardly any fluctuations.
-
Well - My ping time is consistently in the 20s. Thats when things are working well because my MOCA > ONT setup at home adds lots of latency but its very reliable. Not many times is my internet disco.
-
Hey guys,
Our ISP is Verizon Fios. We have a brand new Dell PowerEdge R320 running pfSense 2.1-RC0. The brand of NICs we have are Broadcom…
pinoyboy: So when you say that anything over 3-5 for VoIP is bad, you mean 3-5 ms / %? So in our graph we seem to be between 5 and spike up to 45 which seems terrible in comparison...
Thanks again for all the info!
Ben -
Yes, 3-5%. That's just from our VoIP experience with various clients with high volume calls. Web browsing and other web services do not notice it. Some of the sites have heavy GB FTP sessions concurrently through the day. There should not be any of that red dots kejianshi mentioned, because those are even worse - those are actual connections dropping completely off. For stable VoIP, you also do not want to see those fluctuations jumping up and down. Some providers use (i.e. Comcast) will use consumer grade modem for businesses, but should be using business grade so make sure the equipment they gave you guys fall under business grade too. You as the consumer can make that request if it's a business account. Just because you have large pipe up/down doesn't mean the service will be great for VoIP. It's the quality.
BTW, not fond of those Broadcomm NICS. Even for Virtualization, issues if you use certain hypervisor and/or settings (another topic).
PM and tell me the city you are in. I can perhaps recommend other ITSP to try.
-
More than likely if you on on verizon fios you will never see such a low ping time/latency as pinoyboy. 2-3 is really good.
If you are on verizon fios, you probably have an actiontech router hooked up to coax cable that connects to an ONT (Optical connection) somewhere on or near your house. Between the router and the ONT the connection is MOCA on coax cable and I don't care what anyone says, it is SHIT. Even a typical comcast over cable connection will have much lower latency that will sit around 5 or so, similar to pinoyboy's. The only way you will see a very even RRD quality graph is to patch directly into the gigabit ethernet port on your ONT into your pfsense WAN. A process that breaks your TV service but gives you much better internet.
fios brings a generally faster and theoretically more reliable pipe so long as you don't overwhelm that MOCA link, which is pretty limited. But it seems to introduce about +15ms of latency when idle and ALOT of latency if you are using lots of connection states and bandwidth. Much more so than cable connections.
You see the way pinoyboys's connection in that graph is nice and stable and relatively smooth. His will probably be like that all the time no matter how much connection states / bandwidth he is using. Mine and yours will always look FUGLY because of MOCA… I have no idea what verizon was thinking about, but I tell you, it must have been a decision reached at a corporate board because no engineer worth his salt would have stuck MOCA between a Fiber optic connection and the end user. Its retarded.
-
Hey guys,
I really appreciate all the info you have given me here. It's definitely given me a good place to start in trying to figure out if we are going to be able to fix this issue or if its going to be an ongoing thing.
kejianshi: We have a Fios Business account with no cable service. We have a fiber cable coming directly into the Fios modem which we then plug up with ethernet straight into our pfSense machine. Obviously, I have no idea what they are doing on the other side of that fiber cable so they very well may be doing something that is causing us headaches over here regarding the VoIP.
I have been testing a call all day waiting for it to drop and currently it has been connect for 5 1/2 hours. During that time we have at least 3 of those crappy red dots on our quality graph denoting packet loss, yet the call hasn't dropped yet. Looks like I have a lot of digging and work ahead of me to figure this one out! If you guys have any other revelations I'm all ears :)
Thanks again!
Ben -
Not sure if this will help, look at this - http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
-
Nope - I can tell you I'm on FIOS with an Intel and an Nvidia WAN and they both show same… Latency is generally about 15ms - 20ms but can real quick go to 30 or 40 or 60 if I have a ton of connections going.
Doing the same thing on a few cable setups I have up, the RRD graphs are ALL nice and even around 5. Always.
-
pinoyboy: Nice find. I'll have to give that a shot and see if it helps at all.
kejianshi: We do have a secondary WAN that we use just as a backup. It is a cable connection that is suppose to be 50 down / 5 up. Maybe I should see what kind of quality it has and consider moving my VoIP phones to that connection.
Ben
-
Yeah - It was really annoying for me because I built all the pfsense boxes and set them all up with good parts and good NICS and the one that really, in theory, is supposed to have the best connection consistently has the worse latency (as measured by pinging the gateway).
And verizon charges abit.
They do deliver the most consistent bandwidth but by far the worst latency and ugliest RRD graphs I have seen this side of Nepal or North Korea.
-
I forgot the follow up previously, how many phones / voip endpoints, computers, servers? And you did say this is all under the same subnet - verifying? Are computers daisy chained off the phone's switch on the back; meaning one network connection used by both computer and phone? The Cable service backup you have, is it Verizon? Lastly, to verify, is this the cloud based service of Fonality or the on premise solution?
-
We have about 20 VoIP phones, 40 machines, and 5 servers.
Yes they are all on the same subnet.
We have about 10 of the machines daisy chained to the VoIP phones.
The cable server backup we have is Time Warner. Our Fonality service is cloud based.Thanks
Ben -
This means internal extension to extension calls are also routing to the cloud solution correct? Do you know how many minutes you guys average per month–regardless of before/after Fonality? What is the cost of this solution (question is due to alternative methods)? Does Fonality provide any type of vpn device direct to their network or is this straight through over the Internet (direct over the Internet is least reliable without guaranteed service allocation)? What type of network switch do you have (ALG related question, gigabit, model, QoS parameters, etc)? By the way, these are just subset of information we capture from clients experiencing these issues. You can always PM me if you do not wish to send certain info out publicly.
-
Yes, as I understand it, all calls whether internal or external are routed through the cloud.
We probably use about 1500 mins per month. That's a rough guess based on how many mins we were previously using between our regular calls and our conference bridge system that we now also have with Fonality. I bumped it up a little because now we have more phones then we previously did so I know a few people are making calls that weren't previously because it wasn't convenient.
The cost is roughly $15k per year for all of our phones (we have another office as well that isn't experiencing these issues but their call volume is much much lower). All total we have about 30 phones.
Currently it is just straight through the internet and as far as I know they do not offer any VPN solution. However I will call them tomorrow and ask that question.
We have 2 Netgear GS748TPS switches stacked. These are POE Gigabit switches. http://www.netgear.com/business/products/switches/stackable-smart-switches/GS748TPS.aspx
As far as QOS on the actual switch, I have not touched any of these settings.Let me know if there is any further info I can provide!
Thanks
Ben -
At that price, Fonality is ABSURDLY expensive and the service is not even guaranteed by the way they deliver the lines (no QoS guaranteed over Internet). You can try and see how your other backup cable line is or you can catch up with me anytime during business hours, I sent you a PM.