Asterisk - sip_reg_timeout and Request Sent

[JohnnyBeGood]

Hi all,

I have cable modem connected to pfSense box and when I loose internet with cable modem and after it comes back two trunks on my Elastix PBX (Asterisk v11.13.0 ) cannot re-register. In CLI I get the following:

[2015-11-19 13:53:04] NOTICE[3037]: chan_sip.c:15067 sip_reg_timeout:    -- Registration for '650364952@sip.callwithus.com' timed out, trying again (Attempt #362)
[2015-11-19 13:53:04] NOTICE[3037]: chan_sip.c:15067 sip_reg_timeout:    -- Registration for '5028298@sipgate.de' timed out, trying again (Attempt #362)
[2015-11-19 13:53:24] NOTICE[3037]: chan_sip.c:15067 sip_reg_timeout:    -- Registration for '650364952@sip.callwithus.com' timed out, trying again (Attempt #363)
[2015-11-19 13:53:24] NOTICE[3037]: chan_sip.c:15067 sip_reg_timeout:    -- Registration for '5028298@sipgate.de' timed out, trying again (Attempt #363)
Elastix*CLI> sip show registry
Host                                    dnsmgr Username       Refresh State                Reg.Time
sip.callwithus.com:5060                 N      650364952          120 Request Sent
callcentric.com:5060                    N      17772527263         45 Registered           Thu, 19 Nov 2015 13:53:04
sipgate.de:5060                         N      5028298            120 Request Sent
outgoing.future-nine.com:5060           N      27514245          1185 Registered           Thu, 19 Nov 2015 13:51:10
losangeles.voip.ms:5060                 N      101539             105 Registered           Thu, 19 Nov 2015 13:51:55
5 SIP registrations.
Elastix*CLI>

as you can see all other trunks have registered but callwithus and sipgate have not.

I've contacted support and when this happens on their end they never see anything hitting their servers.
They said that problem is my firewall.
Firewall on Elastix is turned off and only pfSense is left.
Only fix to re-register is to reboot Elastix but lot of times I do not even notice it so no one can call me.

Any help is much appreciated!
pfSense v2.2.5

Here's trunk settings for both trunks if it helps.

CallWithUs

context=from-trunk
host=sip.callwithus.com
qualify=no
username=xxxxxxxxx
secret=xxxxxxxxx
type=friend
insecure=invite

Sipgate

username=xxxxxxx
type=peer
secret=xxxxxx
qualify=yes
insecure=very
host=sipgate.de
fromuser=xxxxxxxx
fromdomain=sipgate.de
dtmfmode=auto
context=from-trunk
authuser=xxxxxxx
registerattempts=0

[awebster]

Couple of things to try…

if you "reload" from asterisk CLI, does this help?

In pfSense, if you reset states (Diagnostics -> States -> Reset States tab), does this help? you might have to do an asterisk CLI reload afterward.

Do either of those providers support SIP/TCP instead of SIP/UDP, that might also help.

In the end what usually works better is a fixed IP with an inbound SIP (+RTP) rule for source IP=trunk provider so that the firewall doesn't drop the state during a WAN outage; usually this results in the behaviour you're seeing.

Last, how are these two "broken" trunks different than the other working trunks?

[JohnnyBeGood]

Hi awebster,

@awebster:

Couple of things to try…
if you "reload" from asterisk CLI, does this help?

Doing reload does not help, only reboot of Elastix.
@awebster:

In pfSense, if you reset states (Diagnostics -> States -> Reset States tab), does this help? you might have to do an asterisk CLI reload afterward.

Pinged sip.callwithus.com and got 208.51.63.146 as IP and did search for that IP in there and removed it and voila trunk registered instantly!
Did same thing for Sipgate and it registered as well.

This is what it showed for that IP:

LAN	udp	208.51.63.146:5060 <- 192.168.1.104:5060	NO_TRAFFIC:SINGLE	
lo0	udp	192.168.1.104:5060 -> 208.51.63.146:5060	SINGLE:NO_TRAFFIC

@awebster:

Do either of those providers support SIP/TCP instead of SIP/UDP, that might also help.

Added tcpenable=yes to my trunk but it did not help with registering.

@awebster:

In the end what usually works better is a fixed IP with an inbound SIP (+RTP) rule for source IP=trunk provider so that the firewall doesn't drop the state during a WAN outage; usually this results in the behaviour you're seeing.

Can you please explain this more? Under DHCP server settings I added "DHCP Static Mappings" for 192.168.1.104 which is my Elastix box.

@awebster:

Last, how are these two "broken" trunks different than the other working trunks?

Here are two examples of trunks that re-register without any issues.

Callcentric

context=from-pstn
fromdomain=callcentric.com
fromuser=1777xxxxxxxx
host=callcentric.com
insecure=port,invite
secret=xxxxxxxxxx
type=peer
defaultuser=1777xxxxxxxx
disallowed_methods=UPDATE
directmedia=no
videosupport=no
disallow=all
allow=g722

Future Nine

username=xxxxxxxxx
type=friend
trustrpid=yes
sendrpid=yes
secret=xxxxxxxx
qualify=no
nat=yes
insecure=port,invite
host=outgoing.future-nine.com
fromuser=xxxxxxxxx
context=from-trunk
canreinvite=no
allow=ulaw

At this point issue is with pfSense stale states. What can I do to leave those alone?

[awebster]

LAN	udp	208.51.63.146:5060 <- 192.168.1.104:5060	NO_TRAFFIC:SINGLE	
lo0	udp	192.168.1.104:5060 -> 208.51.63.146:5060	SINGLE:NO_TRAFFIC

That looks incorrect.
The first state, NO_TRAFFIC:SINGLE means that 192.168.1.104 sent a packet to 208.51.63.146 and go no reply back
The second state for some reason is attached to interface lo0 (loopback), which is wrong, it should be WAN.

Please see this guide: https://doc.pfsense.org/index.php/PBX_VoIP_NAT_How-to

Can you please explain this more? Under DHCP server settings I added "DHCP Static Mappings" for 192.168.1.104 which is my Elastix box.

I was refering to static IP on the WAN side.

[JohnnyBeGood]

@awebster:

Please see this guide: https://doc.pfsense.org/index.php/PBX_VoIP_NAT_How-to

It took me a while but per instructions I've setup all aliases PBX, SIP_Trunks and PBX_Ports.
Now after dozen of modem reboots using its web interface randomly some trunks get stuck with lo0 SINGLE:NO_TRAFFIC state
I just could not see any pattern. Every time I thought I got to work and only one was issue after modem reboot its some other stuck.
What is causing attaching to interface lo0 ?
@awebster:

I was refering to static IP on the WAN side.

Sorry but I don't understand where I would enter static WAN IP?

Thanks for your patience!

[JohnnyBeGood]

So I'm still having same issue and in my quest I found that if I connect Zoiper Android app and try to place the call it disconnects after about 10 seconds. Everything that I found online points back to firewall/pfSense issue.
I was trying to somehow tell not to touch and leave alone traffic from my Elastix box and now I'm stuck with this error that won't go away. Is this something that needs to be submitted as a bug?
I wish there was more detailed explanation how to remove or correct this warning on that screen.

[awebster]

@JohnnyBeGood:

So I'm still having same issue and in my quest I found that if I connect Zoiper Android app and try to place the call it disconnects after about 10 seconds. Everything that I found online points back to firewall/pfSense issue.
I was trying to somehow tell not to touch and leave alone traffic from my Elastix box and now I'm stuck with this error that won't go away. Is this something that needs to be submitted as a bug?

That behaviour you are seeing with zoiper is because an RTP stream cannot be eastablished between the client and the PBX.  Asterisk will detect if it is sending RTP but not receiving RTP and drop the call after 10 seconds.
In order to troubleshoot this sort of problem you need to understand well these things: Networking, Routing, NAT, DHCP, DNS, and SIP/RTP protocol.  Wireshark is a great tool that can help debug VoIP problems quickly, but you have to understand what you're looking at.

@JohnnyBeGood:

I wish there was more detailed explanation how to remove or correct this warning on that screen.

It appears as if the error is coming from a problem in the Aliases definition, either IPs or Ports.  I suggest you check that there is no incorrect data in those Alias definitions.

As far as your original problem is concerned, I don't have any explanation why the state is getting attached to lo0, unless there is something else in your pfSense config that is interfering.

[JohnnyBeGood]

@awebster:

@JohnnyBeGood:

So I'm still having same issue and in my quest I found that if I connect Zoiper Android app and try to place the call it disconnects after about 10 seconds. Everything that I found online points back to firewall/pfSense issue.
I was trying to somehow tell not to touch and leave alone traffic from my Elastix box and now I'm stuck with this error that won't go away. Is this something that needs to be submitted as a bug?

That behaviour you are seeing with zoiper is because an RTP stream cannot be eastablished between the client and the PBX.  Asterisk will detect if it is sending RTP but not receiving RTP and drop the call after 10 seconds.
In order to troubleshoot this sort of problem you need to understand well these things: Networking, Routing, NAT, DHCP, DNS, and SIP/RTP protocol.  Wireshark is a great tool that can help debug VoIP problems quickly, but you have to understand what you're looking at.

@JohnnyBeGood:

I wish there was more detailed explanation how to remove or correct this warning on that screen.

It appears as if the error is coming from a problem in the Aliases definition, either IPs or Ports.  I suggest you check that there is no incorrect data in those Alias definitions.

As far as your original problem is concerned, I don't have any explanation why the state is getting attached to lo0, unless there is something else in your pfSense config that is interfering.

Since this issue is with zoiper for android and PBX how can I "inject" Wireshark in between? I know its possible when its on windows but this is remotely.
As far as Aliases error I deleted all settings and started over and found button that acknowledges all warnings.

[awebster]

@JohnnyBeGood:

Since this issue is with zoiper for android and PBX how can I "inject" Wireshark in between? I know its possible when its on windows but this is remotely.
As far as Aliases error I deleted all settings and started over and found button that acknowledges all warnings.

• SSH to the Elastix box to do this…

• Use which tcpdump to check if tcpdump is installed or not. 
  If it returns /usr/bin/which: no tcpdump in (/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin), then it is not installed.
  If tcpdump is not installed use: yum install tcpdump

• Check what interface Elastix is using with this command: ifconfig -a  Note down which interface (probably eth0) has the Elastix box's IP address.

• Check what IP Asterisk sees your Zoiper client as with this command: asterisk -rx 'sip show peers' .  Note down the IP address corresponding to the zoiper extension number.

• Then run:
    tcpdump -n -i interface -s 0 -w /tmp/capture.cap host zoiperIP
  You will see no output, this is normal.

• Make the test call

• Stop tcpdump with ^C

• Copy the /tmp/capture.cap to a windows box and open it up in wireshark.

• Use the Wireshark Telephony -> Voip Calls tool to analyse the data flow.

You are looking for the SIP/SDP packet where the Zoiper client and Asterisk are negotiating the IP address and port that is going to be used for the RTP traffic.  If you see something unexpected that is the source of the problem.
More details on that here: https://forum.pfsense.org/index.php?topic=101590.msg566835#msg566835

[JohnnyBeGood]

@awebster:

@JohnnyBeGood:

Since this issue is with zoiper for android and PBX how can I "inject" Wireshark in between? I know its possible when its on windows but this is remotely.
As far as Aliases error I deleted all settings and started over and found button that acknowledges all warnings.

• SSH to the Elastix box to do this…

• Use which tcpdump to check if tcpdump is installed or not. 
  If it returns /usr/bin/which: no tcpdump in (/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin), then it is not installed.
  If tcpdump is not installed use: yum install tcpdump

• Check what interface Elastix is using with this command: ifconfig -a  Note down which interface (probably eth0) has the Elastix box's IP address.

• Check what IP Asterisk sees your Zoiper client as with this command: asterisk -rx 'sip show peers' .  Note down the IP address corresponding to the zoiper extension number.

• Then run:
    tcpdump -n -i interface -s 0 -w /tmp/capture.cap host zoiperIP
  You will see no output, this is normal.

• Make the test call

• Stop tcpdump with ^C

• Copy the /tmp/capture.cap to a windows box and open it up in wireshark.

• Use the Wireshark Telephony -> Voip Calls tool to analyse the data flow.

You are looking for the SIP/SDP packet where the Zoiper client and Asterisk are negotiating the IP address and port that is going to be used for the RTP traffic.  If you see something unexpected that is the source of the problem.
More details on that here: https://forum.pfsense.org/index.php?topic=101590.msg566835#msg566835

Ok, so I did not get exact output but it appears that is installed:

[root@localhost ~]# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:21:20.848109 IP 192.168.1.145.ssh > windows7comp.mydomain.net.49437: Flags [P.], seq 2777007169:2777007365, ack 3960846852, win 141, length 196
21:21:20.848302 IP 192.168.1.145.35268 > pfSense.mydomain.net.domain: 56808+ PTR? 100.1.168.192.in-addr.arpa. (44)
21:21:20.848318 IP windows7comp.mydomain.net.49437 > 192.168.1.145.ssh: Flags [.], ack 196, win 253, length 0
21:21:20.848472 IP pfSense.mydomain.net.domain > 192.168.1.145.35268: 56808* 1/0/0 PTR windows7comp.mydomain.net. (78)
21:21:20.848521 IP 192.168.1.145.58282 > pfSense.mydomain.net.domain: 12881+ PTR? 145.1.168.192.in-addr.arpa. (44)
21:21:20.848675 IP pfSense.mydomain.net.domain > 192.168.1.145.58282: 12881 NXDomain* 0/1/0 (103)
21:21:20.848734 IP 192.168.1.145.33881 > pfSense.mydomain.net.domain: 2573+ PTR? 1.1.168.192.in-addr.arpa. (42)
21:21:20.848895 IP pfSense.mydomain.net.domain > 192.168.1.145.33881: 2573* 1/0/0 PTR pfSense.mydomain.net. (75)
21:21:20.849106 IP 192.168.1.145.ssh > windows7comp.mydomain.net.49437: Flags [P.], seq 196:1160, ack 1, win 141, length 964
21:21:20.850110 IP 192.168.1.145.ssh > windows7comp.mydomain.net.49437: Flags [P.], seq 1160:1324, ack 1, win 141, length 164
21:21:20.850286 IP windows7comp.mydomain.net.49437 > 192.168.1.145.ssh: Flags [.], ack 1324, win 256, length 0
21:21:20.851109 IP 192.168.1.145.ssh > windows7comp.mydomain.net.49437: Flags [P.], seq 1324:1600, ack 1, win 141, length 276
21:21:20.852107 IP 192.168.1.145.ssh > windows7comp.mydomain.net.49437: Flags [P.], seq 1600:1764, ack 1, win 141, length 164
21:21:20.852327 IP windows7comp.mydomain.net.49437 > 192.168.1.145.ssh: Flags [.], ack 1764, win 254, length 0

[root@localhost ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:0C:29:DA:FB:XX  
          inet addr:192.168.1.145  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:xxff:feda:fb07/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14565 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25617 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1963719 (1.8 MiB)  TX bytes:9614421 (9.1 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:10811 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10811 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1356014 (1.2 MiB)  TX bytes:1356014 (1.2 MiB)

[root@localhost ~]# 

[root@localhost ~]# asterisk -rx 'sip show peers'
Name/username             Host                                    Dyn Forcerport Comedia    ACL Port     Status      Description                      
100/100                   70.199.128.xxx                           D  Yes        Yes         A  6501     OK (112 ms)                                  
voipms/101xxx_xxxx        96.44.149.186                               Yes        Yes            5060     OK (47 ms)                                   
2 sip peers [Monitored: 2 online, 0 offline Unmonitored: 0 online, 0 offline]
[root@localhost ~]# 

When I make a test call nothing happens with tcpdump but in Asterisk CLI I can see call coming in?

[root@localhost ~]# tcpdump -n -i eth0 -s 0 -w /root/capture.cap host 70.199.128.xxx
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C1780 packets captured
1781 packets received by filter
0 packets dropped by kernel
[root@localhost ~]#

[JohnnyBeGood]

Update:
Looks like i found the problem with Zoiper. I had external port 32920 forwarded to
internal 5060 and as soon as I changed it 5060 to 5060 calls would not disconnect.
Reason for 32920 was because of spammers. What is not working with that forwarding rule?
Note: in Zoiper under ip:port o had pfsense.mydomain.net:32920 but it did not work and after changing to pfsense.mydomain.net:5060 and port forwarding rule it was working.
I'm still having request sent issue with re-register.


[GomezAddams]

I'm not at home right now, so I can't check the details, but your problem rings a bell.

Remember that when a SIP registration takes place, the IP address of the client (your asterisk box in this case) gets sent along in the registration. The host then uses that IP address to try to send data back to the client. Without knowing any better, asterisk will send the IP address of the box it is running on to your SIP provider. Since that IP address is on the inside network, it isn't reachable by the SIP host and communications fail.

There are two ways to fix this: one on the client end, and one on the provider end.

On the client side, asterisk has a setting that will force it to supply the outside IP address of your firewall instead of the local IP. If you don't have a static IP address, you can use a dynamic DNS provider (like DuckDNS) to create a dynamically updated DNS record for your firewall's public IP address and tell asterisk to do DNS lookups on that name every X minutes. Pfsense can handle the dynamic DNS updating for you.

On the SIP provider side, most Internet VoIP providers have realized that the client supplied IP address is likely to be wrong, and ignore it. Instead, they force the registration to use the peer IP address of the incoming connection. Some providers default to this, some you have to ask to turn that option on.

[awebster]

@GomezAddams:

…On the SIP provider side, most Internet VoIP providers have realized that the client supplied IP address is likely to be wrong, and ignore it. Instead, they force the registration to use the peer IP address of the incoming connection. Some providers default to this, some you have to ask to turn that option on.

This recommendation makes the most sense.  Asterisk can become pretty dodgy when using the NAT option to specify which IP it should be putting inside the SIP packets, particularly if you are serving more than one internal subnet.

[JohnnyBeGood]

Thanks for the reply!

@GomezAddams:

On the client side, asterisk has a setting that will force it to supply the outside IP address of your firewall instead of the local IP. If you don't have a static IP address, you can use a dynamic DNS provider (like DuckDNS) to create a dynamically updated DNS record for your firewall's public IP address and tell asterisk to do DNS lookups on that name every X minutes. Pfsense can handle the dynamic DNS updating for you.

On the SIP provider side, most Internet VoIP providers have realized that the client supplied IP address is likely to be wrong, and ignore it. Instead, they force the registration to use the peer IP address of the incoming connection. Some providers default to this, some you have to ask to turn that option on.

I've logged into Unembedded FreePBX inside Elastix and there under Asterisk SIP Settings I already have setup dynamic DNS which is no-ip (Enhanced Dynamic DNS) account registered on pfSense. Will that force asterisk to supply outside IP?

As far as SIP provider. What should I ask them, if they are using my IP from peer details?

[chpalmer]

Do you have any WAN rules pointed at your PBX that would allow your provider to initiate contact with it?

When the firewall comes up and sees traffic from the VOIP provider that appears unsolicited it will block it.

I missed it in your thread if you already answered this.

[JohnnyBeGood]

@chpalmer:

Do you have any WAN rules pointed at your PBX that would allow your provider to initiate contact with it?

When the firewall comes up and sees traffic from the VOIP provider that appears unsolicited it will block it.

I missed it in your thread if you already answered this.

Yes, I do have port forwarding for UDP ports 32920 and 19800-20000 and I have defined those ranges in Asterisk SIP Settings.