Audio is being stripped from RTP stream from VoIP Provider
-
Hi,
I have a somewhat pressing issue where on a very small percentage of calls, we are receiving no audio in the RTP stream. My setup is as follows.
Edge Gateway –-> pfSense (DMZ) ---> PBX
The strange thing with this is that this has only occurred when calls have come from a particular VoIP provider. We are a reseller of VoIP so consequently our customers and us are using the same SIP trunk provider. 99% of calls work fine and we have not seen this issue on any other incoming calls.
My SIP provider has asked for a WAN side capture, which I understand would either prove or disprove the audio reaches us but due to the environment and the setup this isn't really feasible to obtain.
My question is, is it really possible a firewall could be modifying the payload on an incoming RTP stream randomly?
Any thoughts on this would be welcome.
Thank you in advance.
Regards,
Jonathan.
-
My question is, is it really possible a firewall could be modifying the payload on an incoming RTP stream randomly?
No.
This one VoIP provider pretty sure uses other UDP ports to communicate and you haven't opened up that specific range.
But I'm not a VoIP expert at all. -
"due to the environment and the setup this isn't really feasible to obtain."
How is that?? Pfsense can sniff quite easy on any of its interfaces, diag, packet capture.
But NO pfsense is not going to strip data out of a packet that is for sure… Drop the packet ok, modify its contents before sending it on - no..
-
Are there specific group of users for that particular VoIP provider which is behind a different NAT. This seems to be some sort of SIP-ALG issue.
In anyway you should capture the traffic on the interface. I guess the problem is some sort of mismatch between SDP in SIP and where RTP packets sent to.
Per your question, " is it really possible a firewall could be modifying the payload on an incoming RTP stream randomly?"
If pfsense support SIP-ALG then it should be. Also if you are using snort and enabled SIP related categories than snort may drop or block them.
You can capture packets from diagnostics->packet capture. If you send the trace I can look that too.
-
Hi,
Thanks for all the responses. The reason I say it is tricky to capture the WAN traffic is that I am somewhat limited with the packet capture feature built into pfSense. The issue is very intermittent so consequently I would need to start the pcap going and just leave this, possibly for days on end. I do not have space for this to generate such a large pcap file really. I cannot see a way to filter by a range of ports, or to create a ring buffer on pcap files. Maybe somebody could advise different on this?
I can also confirm that I do not have SIP-ALG enabled anywhere, nor am I using an IDS or IPS anywhere in this environment. I am about 95% certain this is a SIP provider issue, but until I can provide a WAN side capture I cannot prove or disprove this. I have checked with the ISP where the PBX is hosted and they cannot so any traffic capture on the edge gateway side so even if I did capture this on the pfSense WAN, the VoIP provider would no doubt then want a pcap from the WAN side of the edge gateway. I understand best practice says I shouldn't double NAT either, but surely I would notice issues with other calls coming in and not just calls from VoIP Provider A at customer site to VoIP Provider A at my site…
To be honest I am thinking of just changing VoIP providers as we have ran into various issues over the years with this particular provider!
Jonathan.
-
While the gui does not allow for a ring file.. If for sure allows you to limit the capture to a protocol and or port. Do you know the source IP/Block of where this traffic coming from this provider would come from you can limit on that.
But you can just use tcpdump directly from cmd line on pfsense and there you can have it ring the files your storing too. You could also mount a share somewhere you have more space and write the files to there. Or just connect a usb disk to pfsense and mount that and write the captures there.
You could also just do a remote capture via ssh to pfsense from some box that has lots of space..
Lots of ways to skin the cat here ;)
-
Thanks for this response, I do indeed know the media servers where the audio is coming from (when we receive it!). There are 2 media servers which I THINK are part of the same subnet, so this should be possible.
It seems this is where my knowledge is somewhat limited! :) I was unaware you could use tcpdump (although thinking about it, it makes sense!) nor did I know I could use SSH for this - this is definitely an option so thank you for this and I'll take a look into this.
I think for this, I need to look at SSH as this issue only manifests itself every few days, it may be 1 in every 300-400 calls. I'll have a play and see what I can come up with. But I suspect even if I can prove the audio doesn't hit the WAN port on the pfSense, they're still going to want a "true" WAN side capture which my ISP is unable to provide.
Anyway thanks for the advice, I'll give it a go and see what I come up with! :)
-
do you have some other firewall in front of pfsense? Does pfsense have a public IP?? If it has a public IP then that would be your sniffing point of your wan..
-
The edge gateway has a LAN IP of 192.168.1.254 /24 and the pfSense has a WAN IP of 192.168.1.1 /24. The LAN side of the pfSense is 172.16.1.254 /24.
-
Ah so your doing nat.. And would not be able to prove that that edge router is not messing with the packets ;)
What is that device is it PE or CE? So your saying you can not sniff on it? Can you put a tap between your wan and its wan and sniff there? Or a switch with span port works, or a old school hub works too if your speeds are not to high..
-
Just a couple questions.
1. What makes you think just the audio is being stripped?
2. Is it being stripped in both directions?A troubleshooting possibility would be to place a phone or phones between the edge gateway and the pfSense box.
Joe
-
Apparently, NAT is quite difficult for VoIP to deal with and it appears you're doing it twice. Voip-info.org is where I go for information and resources for my asterisk PBXs and other VoIP specific info and resources. Something that might help get you going in the right direction is a post on their site that talks about VoIP wth NAT issues.
http://www.voip-info.org/wiki/view/NAT+and+VOIP
Hope that helps.
Joe
-
I am indeed NATing yes, and I am NATing twice… I know NAT can be problematic with VoIP but there is no reason why this shouldn't work. I suppose I could enable outbound static ports on the pfSense and then technically I am only NATing on the edge gateway.
This is all set up in VMware vCloud Director so I have no control of the edge gateway at all, and the functionality is somewhat limited from the portal that I see. I guess the edge gateway would fall into the PE category, however I suspect this is somewhat virtualised and does not physically exist at all.
In response to Joe - We are basically sending the audio out to the VoIP provider from the PBX, we see this leave the PBX and we are not receiving any audio in from the VoIP provider from the caller. I cannot confirm or deny whether or not the audio leaves the WAN port outgoing or reaches the WAN incoming which is proving to be my stumbling block at the minute.
One other thing to note is that when the caller rang, they reported this as a "dead call" as in they didn't even hear the ringing at their side. Now I'm not sure where the 180 ringing originates from but maybe this could provide an insight into something.
-
Also just to add to this, if this WAS a NAT issue would I still be receiving the RTP stream? I still receive the RTP stream when I have no sound on the call. In the last instance, I recieved 1293 RTP packets from the SIP provider, however the payload was all d5's and there was no sound at all. Surely if this was a NAT issue the RTP stream would not be reaching the PBX at all?
-
That's what I'd expect as well but I understand that some of the NAT issues occur even when NAT appears to be working fine. Not to say that's definitely the reason for your problem but apparently you can get mysterious behaviour. Generally speaking there isn't anything I can think of that once set would cause an intermittent issue as infrequent as what you're describing.
Since you say that you have the PBX internally and that's terminated by another company then it's quite possible that you're not actually using SIP across the NAT and we can rule out most of those issues. Asterisk servers are generally setup to use IAX2 which is NAT friendly. However, a one way call is a NAT symptom. The article below seems to explain in more detail how NAT and more specifically PAT can affect your call. Perhaps it will spark some ideas.
https://sipsorcery.wordpress.com/2009/08/05/nat-rtp-and-audio-problems/
Good luck
Joe
-
"I suppose I could enable outbound static ports on the pfSense and then technically I am only NATing on the edge gateway."
No you would still be natting twice you just would not be changing the ports twice with napt. which is how most nat works.. You end up with this, say your talking to a http site
privateIP:52111 –- host-publicIP:80 (nat router) router-publicIP:53222 --- host-publicIP:80
What you have is this
privateIP:52111 --- host-publicIP:80 (nat router) difprivateIP:53222 --- host-publicIP:80 (2nd nat router) router-publicIP:50333 ---- host-publicIP:80
Changing to use static outbound ports does not remove your double with 2 different private IPs you just remove the port number changes.. Which if trying to use static could cause even more issues, when that port is already in use by some other connection when you have multiple hosts that all pick their source port on their own.. You have no way to tell machine X don't use ports X - Y because we want machine B to use them. And now you told the router he can not change the port on the public side to be one that is not in use in the state table because you set it to be static.
-
Also just to add to this, if this WAS a NAT issue would I still be receiving the RTP stream? I still receive the RTP stream when I have no sound on the call. In the last instance, I recieved 1293 RTP packets from the SIP provider, however the payload was all d5's and there was no sound at all. Surely if this was a NAT issue the RTP stream would not be reaching the PBX at all?
RTP is media and SIP is signalling. You can receive the RTP media but your client only listens the IP-port pair which is sent by the SDP in SIP.
So basically, your client may listen NAT IP where as media(RTP) has destination of your WAN IP (or vice versa)
btw, why don't you use a soft SIP client such as linphone and capture the packets from that PC.
-
Voip was not originally built with the idea of NAT. It was a commercial endeavor and not originally marketed at the residential side of things. NAT was added in later.
Below is information my voip provider shows about one of my lines and how its connected. Notice the NATted address is included in the "Contact" line. There is no room for a second NATted address. I can't say it wont work but I don't know how you would make it do so. You might consider using Siproxd package on your pfSense box. It will take out one of the NATs for you. Or at least fool things into thinking that way.
Received:sip:1x4x.2x.2x.x1:5060
Contact:sip:36xxxxxxx9@__172.25.125.20__:5060;user=phone -
"I suppose I could enable outbound static ports on the pfSense and then technically I am only NATing on the edge gateway."
No you would still be natting twice you just would not be changing the ports twice with napt. which is how most nat works.. You end up with this, say your talking to a http site
privateIP:52111 –- host-publicIP:80 (nat router) router-publicIP:53222 --- host-publicIP:80
What you have is this
privateIP:52111 --- host-publicIP:80 (nat router) difprivateIP:53222 --- host-publicIP:80 (2nd nat router) router-publicIP:50333 ---- host-publicIP:80
Changing to use static outbound ports does not remove your double with 2 different private IPs you just remove the port number changes.. Which if trying to use static could cause even more issues, when that port is already in use by some other connection when you have multiple hosts that all pick their source port on their own.. You have no way to tell machine X don't use ports X - Y because we want machine B to use them. And now you told the router he can not change the port on the public side to be one that is not in use in the state table because you set it to be static.
This is very true, I was over looking NAT in this instance. I think for the time being I will leave NAT and outbound NAT as it is as I don't want to make things worse than they already are!
Voip was not originally built with the idea of NAT. It was a commercial endeavor and not originally marketed at the residential side of things. NAT was added in later.
Below is information my voip provider shows about one of my lines and how its connected. Notice the NATted address is included in the "Contact" line. There is no room for a second NATted address. I can't say it wont work but I don't know how you would make it do so. You might consider using Siproxd package on your pfSense box. It will take out one of the NATs for you. Or at least fool things into thinking that way.
Received:sip:1x4x.2x.2x.x1:5060
Contact:sip:36xxxxxxx9@__172.25.125.20__:5060;user=phoneI have read about the Siproxd package, but I am not entirely sure exactly how this works. I will need to do a bit more investigation about this. Thanks for all the advice - it has given me something to go on! :)