VOIP traffic dropped

timlie

Yes, these are even simpler pass rules:

On both just one.

ID Proto Source Port Destination Port Gateway Queue Schedule Description
IPv4 * * * * * none

Packet capture also shows they are talking (where 10.0.0.40 is pbx on ADMINISTRATIE, 10.0.32.30 on nstraat and 10.0.16.30 on kstraat):

21:23:18.859478 IP 10.0.0.40.5060 > 10.0.32.30.5060: UDP, length 282
21:23:18.860787 IP 10.0.0.40.5060 > 10.0.16.30.5060: UDP, length 281
21:23:18.862301 IP 10.0.16.30.5060 > 10.0.0.40.5060: UDP, length 255
21:23:18.867157 IP 10.0.32.30.5060 > 10.0.0.40.5060: UDP, length 256
21:23:40.953233 IP 10.0.32.30.5060 > 10.0.0.40.5060: UDP, length 282
21:23:40.960329 IP 10.0.0.40.5060 > 10.0.32.30.5060: UDP, length 257
21:23:44.417042 ARP, Request who-has 10.0.0.40 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
21:23:45.408131 ARP, Request who-has 10.0.0.40 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
21:23:52.409669 IP 10.0.0.40.5060 > 10.0.32.30.5060: UDP, length 282
21:23:52.410951 IP 10.0.0.40.5060 > 10.0.16.30.5060: UDP, length 281
21:23:52.412467 IP 10.0.16.30.5060 > 10.0.0.40.5060: UDP, length 255
21:23:52.417268 IP 10.0.32.30.5060 > 10.0.0.40.5060: UDP, length 256

awebster

VoIP calls have 2 separate components; SIP for establishing the call, running on port 5060, and RTP for transporting the audio, running on dynamically allocated UDP ports >10000 typically. If calls are ringing, but there is no audio path, I'd suspect an RTP problem.

Make sure the PBXes are not configured to think that they are behind NAT, this is a PBX setting, and it will confuse things.
You can try tcpdump -s 0 -n -i <interface>-w <some_file>host <ip_of_pbx>and place a call.
<interface>should be the inteface on which the PBX is connected, but you can also try it on some of the other interfaces to be sure the traffic is passing as expected.

Use wireshark to open the resulting capture files, and use the Telephony -> VoIP calls option to analyse what's going on.

This site has some decent diagrams of how SIP works.
http://www.in2eps.com/fo-sip/tk-fo-sip-ex3261.html</interface></ip_of_pbx></some_file></interface>

timlie

Thanks awebster, i'll try this tomorrow!

timlie

I did some testing with tcpdump.

As you can see in the wireshark screenshot, there is sip communication but there is no rtp traffic.

This is happening all the time but I have no idea why this rtp traffic does not come through…

Searching the internet, most of the time they say it's a NAT issue with port modification which RTP tends to have difficulties with.
Although in our situation there is no NAT involved and as far as I know also no port modification (or do I see this wrong?).

awebster

Look inside the SIP/SDP packets.
From 10.0.32.30 to 10.0.0.40 there is a SIP INVITE with an SDP.
Inside you'll see m=audio ##### RTP/AVP 0 8 101 or something similar. ##### is the port number that RTP is going to use.
Likewise in the SIP OK from 10.0.0.40 to 10.0.32.30 there is an SDP, and it will have similar information.
First, make sure that the port numbers can pass through the firewall, although, because you are seeing no RTP traffic whatsoever, I'd suspect a codec mismatch.

What is really important is that both sides can agree on what codec to use.
In North America, it is ulaw, in which case, you should also have a=rtpmap:0 PCMU/8000 in the SDP packet; on both sides.
In the rest of the world, it is typically alaw, so you'll see a=rtpmap:8 PCMA/8000.
In either case, both sides must both present a common set of codecs, otherwise they won't be able to communicate.

timlie

Thanks awebster!

In the SDP from the invite is the following information:

Session Description Protocol
Session Description Protocol Version (v): 0
Owner/Creator, Session Id (o): Telrad_Connegy 1077972984 1077972984 IN IP4 10.0.32.30
Owner Username: Telrad_Connegy
Session ID: 1077972984
Session Version: 1077972984
Owner Network Type: IN
Owner Address Type: IP4
Owner Address: 10.0.32.30
Session Name (s): SIP Call
Connection Information : IN IP4 10.0.32.31
Connection Network Type: IN
Connection Address Type: IP4
Connection Address: 10.0.32.31
Session Attribute (a): connegy_back_to_back:{ <tail_ssrc a048000=""><tail_sys_id a=""><tail_nat_num 8=""><tail_port_nat_val 0=""><tail_behind_nat 0=""><tail_ssrc_type 0=""><tail_profile_type ff="">}
Session Attribute Fieldname: connegy_back_to_back
Session Attribute Value: { <tail_ssrc a048000=""><tail_sys_id a=""><tail_nat_num 8=""><tail_port_nat_val 0=""><tail_behind_nat 0=""><tail_ssrc_type 0=""><tail_profile_type ff="">}
Time Description, active time (t): 0 0
Session Start Time: 0
Session Stop Time: 0
Media Description, name and address (m): audio 6000 RTP/AVP 18 101
Media Type: audio
Media Port: 6000
Media Protocol: RTP/AVP
Media Format: ITU-T G.729
Media Format: DynamicRTP-Type-101
Media Attribute (a): rtpmap:18 G729/8000
Media Attribute Fieldname: rtpmap
Mdia Format: 18
MIME Type: G729
Sample Rate: 8000
Media Attribute (a): rtpmap:101 telephone-event/8000
Media Attribute Fieldname: rtpmap
Media Format: 101
MIME Type: telephone-event
Sample Rate: 8000

For the SIP OK this is the SDP

Session Description Protocol
Session Description Protocol Version (v): 0
Owner/Creator, Session Id (o): Telrad_Connegy 1077953016 1077953016 IN IP4 10.0.0.40
Owner Username: Telrad_Connegy
Session ID: 1077953016
Session Version: 1077953016
Owner Network Type: IN
Owner Address Type: IP4
Owner Address: 10.0.0.40
Session Name (s): SIP Call
Connection Information (c): IN IP4 10.0.0.41
Connection Network Type: IN
Connection Address Type: IP4
Connection Address: 10.0.0.41
Session Attribute (a): connegy_back_to_back:{ <tail_ssrc 9118001=""><tail_sys_id 9=""><tail_nat_num 8=""><tail_port_nat_val 0=""><tail_behind_nat 0=""><tail_ssrc_type 0=""><tail_profile_type ff="">}
Session Attribute Fieldname: connegy_back_to_back
Session Attribute Value: { <tail_ssrc 9118001=""><tail_sys_id 9=""><tail_nat_num 8=""><tail_port_nat_val 0=""><tail_behind_nat 0=""><tail_ssrc_type 0=""><tail_profile_type ff="">}
Time Description, active time (t): 0 0
Session Start Time: 0
Session Stop Time: 0
Media Description, name and address (m): audio 6000 RTP/AVP 18 101
Media Type: audio
Media Port: 6000
Media Protocol: RTP/AVP
Media Format: ITU-T G.729
Media Format: DynamicRTP-Type-101
Media Attribute (a): rtpmap:18 G729/8000
Media Attribute Fieldname: rtpmap
Media Format: 18
MIME Type: G729
Sample Rate: 8000
Media Attribute (a): rtpmap:101 telephone-event/8000
Media Attribute Fieldname: rtpmap
Media Format: 101
MIME Type: telephone-event
Sample Rate: 8000

Summary

10.0.32.30 has audio 6000 RTP/AVP 18 101
10.0.0.40 has audio 6000 RTP/AVP 18 101

But I can see this:

Connection Information (c): IN IP4 10.0.32.31
Connection Information (c): IN IP4 10.0.0.41

Which now for me makes sense, these PBX's have two interfaces, the first one for the data link, the second for the voice.

So it seems that rtp traffic is not "starting" from these second interfaces.</tail_profile_type ></tail_ssrc_type ></tail_behind_nat ></tail_port_nat_val ></tail_nat_num ></tail_sys_id ></tail_ssrc ></tail_profile_type ></tail_ssrc_type ></tail_behind_nat ></tail_port_nat_val ></tail_nat_num ></tail_sys_id ></tail_ssrc ></tail_profile_type ></tail_ssrc_type ></tail_behind_nat ></tail_port_nat_val ></tail_nat_num ></tail_sys_id ></tail_ssrc ></tail_profile_type ></tail_ssrc_type ></tail_behind_nat ></tail_port_nat_val ></tail_nat_num ></tail_sys_id ></tail_ssrc >

awebster

Running voice and data on different physical interfaces is a bit strange, and would certainly be difficult for any routing table to work properly to figure out which port to use to send packets to the destination subnet.
If the PBX (what brand?) supports it, use just one interface.

timlie

Is something old (Telrad) installed by third party.
Doubt that it will be possible to use only one interface.

awebster

So what does the routing table look like on that platform?
How do you enter a static route, and can you choose the interface on which it applies? Maybe you need 2 routes, one for each interface?

timlie

For 10.0.0.40 and 41 it uses our interface card (10.0.0.1) on ADMINISTRATIE as gateway.
For 10.0.16.30 and 31 on KSTRAAT there is a static route on the gateway which sends traffic back to our 10.0.0.0/20 network.

These routes are correct because pinging between eachother works.

awebster

@timlie:

For 10.0.0.40 and 41 it uses our interface card (10.0.0.1) on ADMINISTRATIE as gateway.
For 10.0.16.30 and 31 on KSTRAAT there is a static route on the gateway which sends traffic back to our 10.0.0.0/20 network.

These routes are correct because pinging between eachother works.

Right, but the ping is only using one of the two interfaces on the PBX, unless you can specify the source IP when initiating the ping.
I suggest you use port mirroring on your switches to try and capture the traffic from each of the interfaces on the PBX to see if there actually is an attempt to send RTP traffic on either of these interfaces, and to what IP it is trying to send it, and more importantly to what L2 MAC address it is trying to deliver it.

timlie

Ok, I think we should indeed try that in two weeks (we have a week holiday here next week). Thanks for your advice awebster, really appreciate this!

timlie

I was thinking, would it be possible that I have to use "Bypass firewall rules for traffic on the same interface"?

Myself, I guess not because there is a static route on the other side but traffic always comes back on an interface on our pfsense…

awebster

Bypass firewall rules for traffic on the same interface only applies if you have a situation where traffic is bouncing off the interface (typically LAN side).
IE: You have multiple different subnets that are reachable through a router on the LAN side, then you don't want the firewall inspecting traffic that is staying "inside" the network.

Usually in this type of setup, you would have all hosts pointing to an internal router, whose default gateway is the pfSense LAN interface, but in some cases you might have some hosts that are pointing to pfSense LAN interface as default gateway and they need to reach other hosts via the router that is connected to pfSense LAN interface.

timlie

Ok, thanks. I'll get it.

That's not the case in our situation.

timlie

Tomorrow I can test again on this case.
I received some extra information on these pbx's.

Apparantly the data signals are send on the first interface of the pbx (10.0.0.40 ; 10.0.16.30 ; 10.0.32.30) where the voice (RTP) is send over the second interface of the pbx (10.0.0.41 ; 10.0.16.31 ; 10.0.32.31).

The data signals are send correct between pbx's as seen in the capture. Voice is not.

Would it be possible that the synchronous RTP communication is blocked because there is no firewall state created for the second interfaces?

Thanks!

awebster

@timlie, the state gets created when the first packet is sent, if the firewall allows it. and the firewall rules that you've shared earlier seem to indicate that everything open on the subnet, so RTP should be passing.

At this point everything is pointing to a routing problem on the PBX's second interface.
Here is one way to troubleshoot that:
On the pfSense that is working with PBX#1, if you run the following from the command prompt:
tcpdump -s 0 -n -i lan_interface host PBX#1_IP#1 or host PBX#1_IP#2 or host PBX#2_IP#1 or host PBX#2_IP#2
you should see communications taking place, including RTP if any.
You can optionally add -w filename.cap to the tcpdump command to write it to a file, and then open it up in wireshark.

If you see RTP traffic, then run the same command on the pfSense that is working with PBX#2 and check that the RTP traffic that PBX#1 is sending is going to PBX#2.

timlie

Thanks awebster,

This capture showed us that the second interface was still using our old gateway because of an ARP table which this pbx didn't refresh.

Fixed now!
Thanks a lot!

awebster

Glad that was easy to fix.
Please consider editing your original post to add [SOLVED] to the subject.