UDP packets dropping is preventing 2-way VOIP call over an OpenVPN tunnel
I have a tun VPN setup between a main office running pfSense 2.0 and a remote one running tomato 1.27vpn. Almost everything works perfectly, on both sides of the tunnel clients can see each other, but the VOIP phones that I have at the remote sites can not establish a 2-way call. They are able to talk to the PBX using management ports and register with it successfully and when they receive a call the user on the remote side can hear the caller, but not the other way.
dev ovpns7 dev-type tun dev-node /dev/tun7 writepid /var/run/openvpn_server7.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher BF-CBC up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local xxx.xxx.xxx.xxx tls-server server 10.8.5.0 255.255.255.0 client-config-dir /var/etc/openvpn-csc tls-verify /var/etc/openvpn/server7.tls-verify.php lport 1199 management /var/etc/openvpn/server7.sock unix push "route 10.0.0.0 255.255.0.0" push "dhcp-option DNS 10.0.0.2" ca /var/etc/openvpn/server7.ca cert /var/etc/openvpn/server7.cert key /var/etc/openvpn/server7.key dh /etc/dh-parameters.2048 tls-auth /var/etc/openvpn/server7.tls-auth 0 comp-lzo persist-remote-ip float route 10.4.0.0 255.255.255.0
Client specific overrides on the server:
iroute 10.4.0.0 255.255.255.0 push "route 10.0.0.0 255.255.0.0 10.8.5.1"
daemon client dev tun11 proto udp remote xxx.xxx.xxx.xxx 1199 resolv-retry 30 nobind persist-key persist-tun comp-lzo adaptive cipher BF-CBC verb 3 script-security 2 up updown.sh down updown.sh tls-auth static.key 1 ca ca.crt cert client.crt key client.key status-version 2 status status
If I do a packet capture on the server I get the following
00:58:11.364298 AF IPv4 (2), length 232: (tos 0xc0, ttl 64, id 2372, offset 0, flags [none], proto ICMP (1), length 228) 10.8.5.6 > 10.0.0.141: ICMP 10.8.5.6 udp port 8000 unreachable, length 208 (tos 0x0, ttl 127, id 0, offset 0, flags [DF], proto UDP (17), length 200) 10.0.0.141.12124 > 10.8.5.6.8000: [udp sum ok] UDP, length 172 MPLS extension v15 packet not supported
and for the same call on the client I get
Jan 29 00:58:56 unknown user.warn kernel: ACCEPT IN=br0 OUT=tun11 SRC=10.4.0.112 DST=10.0.0.141 LEN=200 TOS=0x04 PREC=0x00 TTL=63 ID=16158 PROTO=UDP SPT=8000 DPT=8000 LEN=180
If I run nmap against the PBX addresses I can see the ports 1718 and 1720 (H.323) as open, so I'm thinking that there's a problem routing UDP traffic through the VPN.
The main site router has the following firewall rule on the WAN interface
Proto Source Port Destination Port Gateway Queue Schedule Description UDP * * WAN address 1199 * none 10.4.0.0/24 - OpenVPN
and a rule generated by the wizard for the OpenVPN
Proto Source Port Destination Port Gateway Queue Schedule Description * * * * * * none OpenVPN wizard
I'm completely stuck now, I've tried switching to TAP but that didn't work, tried a dozen or so different client specific overrides and custom configuration options with all of the possible permutations, all with the same outcome. I'd really appreciate any help that anyone can offer.
You can ping no problem across the VPN?
This is indicative of what the issue is:
00:58:11.364298 AF IPv4 (2), length 232: (tos 0xc0, ttl 64, id 2372, offset 0, flags [none], proto ICMP (1), length 228)
10.8.5.6 > 10.0.0.141: ICMP 10.8.5.6 udp port 8000 unreachable, length 208
Port unreachable either means the target device is rejecting the traffic, or something that's filtering is rejecting the traffic. I'm guessing 8000 was your RTP port in that case, but not enough there to tell that for sure. Usually RTP would be in the range of 10000-20000, maybe the phone is only accepting RTP in that range and rejecting 8000. Maybe the firewall rules on Tomato are rejecting 8000. Could be something else in the network.
The PBX has 3 IP addresses, 1 is on a SIP trunking card (10.0.0.142) and the other two are on the processor board (10.0.0.140 & 141), I can ping all 3 from within the LAN and from over the VPN.
Your guess is correct, 8000 is the first of 64 RTP ports. I don't know if I mentioned this but the phones do work when plugged into the local network.
I had considered Tomato might be rejecting traffic, but the
10.8.5.6 > 10.0.0.141: ICMP 10.8.5.6 udp port 8000 unreachable
led me to believe that it was on the pfSense side.
IIRC Tomato is pretty limited in what it lets you do, though I believe the last time I worked on it I was able to upload a tcpdump binary to it for troubleshooting purposes (it wasn't included). You'll need to trace that unreachable back to find out where it's being generated. I would start on the br0 interface (should be "LAN") on Tomato and see what it's showing. If you're getting the unreachable back there sourced from the phone's MAC, it's coming straight from the phone. If you're not, keep moving your capture to different reference points until you find what's generating it.
Thanks for the advice cmb. I loaded tcpdump into my tomato router and when I try to make a call I get several packets that look like this one showing up on the OpenVPN tunnel
52 3.917411 10.8.5.6 10.0.0.141 ICMP 244 Destination unreachable (Port unreachable)[Packet size limited during capture]
all of which use 12244 as the source port and 8000 as the destination, which matches the capture that I had done on my pfSense router.
I could be wrong, but if this traffic is making it out of the LAN interface on the remote site and to the VPN interface where for some reason they are denied it suggests to me that the router at the other end is blocking it. There is only a firewall rule to allow UDP WAN traffic on port 1199, as was mentioned previously.
Next I ran a capture on the WAN interface of the pfSense router and everything looks fine there, but then when I ran another on the LAN interface I get the same packets as before showing destination unreachable, along with several other successful packets like these:
196 4.040371 10.8.5.6 10.0.0.141 UDP 214 Source port: 1024 Destination port: 12026 197 4.041143 10.0.0.141 10.8.5.6 UDP 214 Source port: 12026 Destination port: irdmi
Same thing show up when you tcpdump on br0? (or whatever the LAN-facing interface is on Tomato, IIRC it's br0). If it shows up on br0, it's being generated by the phone. If it only shows up on the tunnel interface on that side, it's being generated by Tomato.
No, not on br0 (that is the LAN interface), only on the tunnel interface. So you're saying in that case that the Tomato router is generating the unreachable destination ICMP packet? If that's the case would they also be seen on the pfSense side? Because as mentioned, I do see them in the capture performed on its LAN interface.
From what you've shown, it has to be from either the phone or Tomato. If you don't see it on the LAN of Tomato and do on the tunnel, then it's Tomato itself generating that. The other alternative when you're seeing it there on the tunnel would be the phone sending it, in which case you'd see it on br0 too. Not seeing it on br0 eliminates the phone. I would presume the cause to be iptables rejecting it, and that's probably the most likely, but I only know enough about Tomato to be dangerous. There may be other things in it that can reject traffic.
The Tomato iptables are
Chain INPUT (policy DROP) target prot opt source destination ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 DROP 0 -- 0.0.0.0/0 x.x.x.x DROP 0 -- 0.0.0.0/0 0.0.0.0/0 state INVALID ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy DROP) target prot opt source destination ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 DROP 0 -- 0.0.0.0/0 0.0.0.0/0 state INVALID TCPMSS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 tcpmss match 1461:65535 TCPMSS set 1460 ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED wanin 0 -- 0.0.0.0/0 0.0.0.0/0 wanout 0 -- 0.0.0.0/0 0.0.0.0/0 ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 upnp 0 -- 0.0.0.0/0 0.0.0.0/0 ACCEPT 0 -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED ACCEPT 0 -- 10.8.5.0/24 0.0.0.0/0 REJECT 0 -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain upnp (1 references) target prot opt source destination Chain wanin (1 references) target prot opt source destination ACCEPT tcp -- 0.0.0.0/0 10.4.0.107 tcp dpt:47888 ACCEPT udp -- 0.0.0.0/0 10.4.0.107 udp dpt:47888 ACCEPT tcp -- 0.0.0.0/0 10.4.0.107 tcp dpt:47808 ACCEPT udp -- 0.0.0.0/0 10.4.0.107 udp dpt:47808 ACCEPT udp -- 0.0.0.0/0 10.4.0.1 udp dpt:1199 Chain wanout (1 references) target prot opt source destination
and now that I look at them the rule for "reject-with icmp-port-unreachable" stands out. Unfortunately I had to take the phone back to work so I'll have to bring another one home tomorrow to test it after removing that rule.
Thanks to cmb this is now working. The trick was to change the mode from Automatic to Custom in VPN Tunneling->Client->Firewall, and add the following iptables rules under Administration->Scripts->Firewall:
# apply forwarding for OpenVPN Tunneling iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A FORWARD -s 10.8.5.0/24 -j ACCEPT iptables -A FORWARD -s 10.0.0.0/16 -j ACCEPT iptables -A FORWARD -j REJECT # enable forwarding echo 1 > /proc/sys/net/ipv4/ip_forward
Now I just have to figure out how to mark this thread as solved.