SOLVED: Having a maddening time getting a SIP Codec to work correctly.



  • I have two audio-over-IP devices called the Comrex Access.  One is a rackmount, the other a portable version you can take on the road.

    Right now they are both behind a pfsense hardware router.

    UDP on ports 5060, 6014, and 6015 are being forwarded to the rackmount.

    I've created static ports as well for UDP from the rackmount.

    The device is unable to connect properly to a different audio-over-IP device (brand is Tieline) that is set into SIP mode at a radio station.  I say properly because the Comrex does connect, however no audio data is transmitted to or from the Tieline.

    For the sake of testing, I hooked up my Comrex Access portable unit and flipped the forwards and static ports to it.  It was able to connect properly to the Tieline.

    One strange difference between the two Comrex units, is the rackmount's status readout reports that its NAT TYPE is "Port Fwd." while the portable unit says its is "Symmetric".  Both should have identical configurations and are connected to the same router, and I don't know enough about networking to ascertain why the different NAT TYPEs are showing up.

    I've looked and looked and tried various configurations but nothing can get the connection to the Tieline on the rackmount to work.

    That alone is frustrating, but worse, the Comrex is supposed to allow for incoming connections from SIP clients.  Well, the incoming calls connect, but no audio data is sent or received by the Comrex (on both the rackmount and portable - when ports are supposedly being properly forwarded).  As a test, I disconnected my cable modem from the pfsense and plugged it directly into the Comrex portable unit.  Connecting with a SIP client was no problem then, so it's pretty clear that pfsense is doing something to my packets.

    I've now spent several hours trying to troubleshoot this, so I'm coming to you for help.  Any suggestions?  :(



  • Talked with Comrex Tech support today and they said my load balancing could be causing the issue.  I have two internet connections here.  I'll see if I can switch to failover, or is there a way to make sure certain LAN IPs are not subject to load balancing?



  • I believe I've figured out how to set my Comrex unit to avoid the load balancer, via the LAN Firewall Rules - I made one that goes above the load balancing rule that drives all the Comrex sourced traffic through my Time Warner connection.

    However, this has still not solved the problem.

    I spoke again with Comrex today and they suggested forwarding TCP in addition to the required UDP on ports 6014 and 6015 so a port checker would be able to see it if it was working.

    I did that and when we checked it, the ports were coming back as closed.  Prior to that the ShieldsUP scanner listed them as "Stealth"

    So, now it seems like my port forward isn't working properly and I'm not sure why.



  • Post screens of your NAT port forward and WAN firewall rules, sanitized of public details of course.  You an also go through this list and verify all is proper:

    https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting



  • Thanks for the reply.

    Here are the port forwards, which include 5060, 6014, and 6015 - what their SIP compatibility requires be forwarded to the Comrex unit.

    Also, the WAN rules that were created automatically when the port forwards we made.

    Finally, the LAN rules that I made to avoid load balancing for the Comrex unit.

    Thanks for helping this non-IT-pro.








  • Your NATs look OK unless I've missed something.  I would temporarily disable pfBlocker and test again.



  • I'll try that, but pfBlocker was installed after this problem started.  I just happened to discover that package as I was trying to figure all this out.

    Just tried disabling it, but no change.

    The radio station unit is no longer available for testing, so I'm using a program called Linphone (it's an Android SIP client) to connect to the Comrex.  As before, it connects, but no data is transmitted or received by the Comrex.

    I know the Linphone is setup correctly because the Comrex company has one of their units on the public internet that I can connect to with Linphone.  Linphone is on 4G, so not the same network as the Comrex, which is on my Time Warner WAN.

    I'm attaching two screenshots of the states that are active for the IP of my cell phone while I connect to the Comrex through PFSense router via Linphone.  I connect successfully, but no audio data is transferred, and after 30 seconds or so, Linphone crashes.

    When I capture packets associated with the Cell Phone IP it shows packets leaving my WAN IP from port 6014 and going to the Cell Phone IP port 7076.

    When I use my Comrex to connect to Comrex's test unit via SIP, it works fine.  Packet capture shows packets leaving and coming, states show multiple states on ports 6014 and 6015 whereas as you can see when I dial into my Comrex from Linphone those ports show single/no traffic.

    I'm so confused.  Thanks for any insights and suggestions you might have!






  • Anything related in the firewall log being blocked on WAN?

    I use Polycom VoIP phones here and they just work without any voodoo.  No port forwards required.  They reach out and keep the states open for inbound signalling and data.



  • No entries for the Cell Phone IP in the firewall logs when I connect via Linphone.

    If I turn off the port forwards, firewall blocks to the Cell Phone IP show up on port 5060, as they should.



  • It's probably an advanced network option like strict NAT or static source ports or something like that.  Search these forums for keywords like SIP, VoIP, no audio as I've seen these types of cases before but I haven't paid them much attention.



  • Destination on your WAN firewall rules should be the IP address of the Comrex box.. or the IP subnet that covers multiple boxes.

    Get rid of any port forwarding and see how that works out for you.

    Broadcast stuff we use is all T-1 so this is out of the knowledge base. But ROIP and VOIP is usually visited once a week or so.



  • It is.  "Comrex_Access_Rack" is an alias for the LAN IP for the Comrex box.





  • @chpalmer:

    Destination on your WAN firewall rules should be the IP address of the Comrex box.. or the IP subnet that covers multiple boxes.

    Get rid of any port forwarding and see how that works out for you.

    Getting rid of the port forward doesn't make sense, and doesn't work - that just makes Linphone not connect at all.



  • @FTL_Ian:

    Getting rid of the port forward doesn't make sense, and doesn't work - that just makes Linphone not connect at all.

    Just a test.

    In the SIP world it does make sense because the client SIP device reports in its header its NAT address.



  • I did get rid of the 6014 and 6015 port forwards and that made no difference.  It still connects, but no data is transferred.

    I've tried all the troubleshooting links I can find.  There must be something that will work, but I'm baffled as to what it is.  Having spent nearly two weeks on this, I'm getting close to contacting support.


  • Netgate

    Have any documentation from Comrex describing exactly what they need from a firewall - NAT in particular?



  • They say to forward ports 5060, 6014, and 6015 UDP to the unit and it needs a static IP.  I've since assigned the unit to avoid load balancing as I'm on a dual-WAN, but that didn't solve the problem.

    Here's their guide to connect Linphone SIP client with the Comrex:
    http://www.comrex.com/wp-content/uploads/2016/01/Linphone-technote-for-ACCESS-and-BRIC-Link.pdf

    Works fine when I connect to their test, works fine when I plug the Comrex directly into my cable modem, so it's definitely not a problem with Linphone or the Comrex - seems like something in PFsense is getting in the way.


  • Netgate

    That's great but there is nothing to get in the way there.

    I see no mention of outbound connections from the codec so delete any static port outbound NAT you have created. It won't help and might break something else.

    What are the contents of all your aliases?

    I see no mention of TCP in that document. Why is your SIP forward TCP/UDP?

    You do not need to worry about Multi-WAN on inbound connections from clients. pf reply-to handles that and is automatic as long as that connection is set up as a WAN (Has a gateway set on the interface). That is unless the default gateway on the codec is not set to pfSense LAN in which case you'll have all sorts of asymmetric routing issues.


  • Netgate Administrator

    This right?
    http://www.comrex.com/wp-content/uploads/2016/02/ACCESS-Rack-Manual.pdf

    Are you running v3.0 firmware? If not then you need to be forwarding different ports.

    In the case of SIP, this
    must be three discrete ports (For Comrex codecs these are UDP 5060, 5014
    and 5015)
    <6014 and 6015 with 3.0 firmware>

    Do you see any blocked traffic in the firewall log from the client IP you are trying to connect from?

    Steve



  • Derelict:

    I've tried static port on and off and neither worked, so it's off at this time.

    I was forwarding TCP so a port checker would be able to verify the forwards.  It's not necessary for the Comrex.  Only UDP.

    Do you want the full contents of my aliases on ports, or just those related to the Comrex?



  • @stephenw10:

    This right?
    http://www.comrex.com/wp-content/uploads/2016/02/ACCESS-Rack-Manual.pdf

    Are you running v3.0 firmware? If not then you need to be forwarding different ports.

    In the case of SIP, this
    must be three discrete ports (For Comrex codecs these are UDP 5060, 5014
    and 5015)
    <6014 and 6015 with 3.0 firmware>

    Do you see any blocked traffic in the firewall log from the client IP you are trying to connect from?

    Steve

    I'm running Comrex 4.0-p9 - the latest firmware.  Ports 5060 and 6014 and 6015 are the proper ports, per the advanced settings in the unit.

    I've checked the firewall, and no, I don't see any blocked traffic from the IP in question.



  • As you can see in the pic I posted in this post: https://forum.pfsense.org/index.php?topic=121139.msg671629#msg671629

    There are states between my cell phone IP and all three ports in question.  But no meaningful amount of audio data (like a 64kbps audio stream) is being transmitted.



  • (Going on the radio for a few hours - will check back later.  Thanks for the help, all.)


  • Netgate Administrator

    Ok, I would try grabbing a packet capture on the internal interface and filter by the IP of the Comrex unit.

    If you can inspect the SIP packets in wireshark I'm betting that it's handing out it's internal private IP as the destination for the RTP traffic.

    There appears to be a setting in the Comrex to force that to the external IP:

    Under
    Advanced System Settings
    , a field is available called
    Public IP Override
    . Any address put into that field will be pasted into the
    address SIP field

    Steve


  • Netgate

    There simply doesn't look to be anything special about this unit. No special NAT requirements.

    Just need the contents of the aliases specific to the Comrex.

    From those states it looks like the phone IP connects inbound on 5060 and then the Comrex is attempting to connect outbound on 6014 and 6015 to the Cell Phone's IP address and is receiving nothing in reply.

    Allowing that traffic will be up to the firewall at the cell phone side. Those counters show the outbound traffic, with zeroes in reply and those two captures are a good example of what is meant by static source port.

    You sure you have this all configured correctly?

    It looks to me like the "server" unit should tell the phone unit to connect back to it on 6014 and 6015 but, instead, it is just trying to connect outbound sourced from 6014/6015. Destination 7076 and 7077 in both examples.



  • @stephenw10:

    Ok, I would try grabbing a packet capture on the internal interface and filter by the IP of the Comrex unit.

    If you can inspect the SIP packets in wireshark I'm betting that it's handing out it's internal private IP as the destination for the RTP traffic.

    There appears to be a setting in the Comrex to force that to the external IP:

    Under
    Advanced System Settings
    , a field is available called
    Public IP Override
    . Any address put into that field will be pasted into the
    address SIP field

    Steve

    Hrm, here's the top of the packet capture I ran during a test connection from Linphone.  The source and destination ports are:
    line 1: 37524 / 5060
    2:  5060 / 37524
    3: 1783 / 7077
    4-100: 61245 / 7076

    Searching for the cell phone IP in the firewall logs of pfsense shows nothing - no pass, no block.  Shouldn't it show passes?



  • Netgate

    Under
    Advanced System Settings
    , a field is available called
    Public IP Override
    . Any address put into that field will be pasted into the
    address SIP field

    Did you do this?

    No. Pass rules do not log unless you explicitly enable that on the rule.

    Again, that shows good two-way SIP initiated by the Phone IP followed by OUTBOUND traffic to the Phone IP on ports 7076 and 7077. That will have to be passed at the Phone IP side.



  • @Derelict:

    There simply doesn't look to be anything special about this unit. No special NAT requirements.

    Just need the contents of the aliases specific to the Comrex.

    From those states it looks like the phone IP connects inbound on 5060 and then the Comrex is attempting to connect outbound on 6014 and 6015 to the Cell Phone's IP address and is receiving nothing in reply.

    Allowing that traffic will be up to the firewall at the cell phone side. Those counters show the outbound traffic, with zeroes in reply and those two captures are a good example of what is meant by static source port.

    You sure you have this all configured correctly?

    It looks to me like the "server" unit should tell the phone unit to connect back to it on 6014 and 6015 but, instead, it is just trying to connect outbound sourced from 6014/6015. Destination 7076 and 7077 in both examples.

    I've attached the comrex port aliases, per your request.  The 10000:30000 range is new - I added them per PFsense's official support person's request, though they are not required by Comrex.  Adding them made no difference.

    I believe I have it configured correctly - that's what PFsense's support person told me.  That's what one of the earlier respondents said here on this thread, but clearly I've got something set wrong somewhere, as the Comrex works fine when it's plugged directly into the cable modem.




  • @Derelict:

    Under
    Advanced System Settings
    , a field is available called
    Public IP Override
    . Any address put into that field will be pasted into the
    address SIP field

    Did you do this?

    No. Pass rules do not log unless you explicitly enable that on the rule.

    Again, that shows good two-way SIP initiated by the Phone IP followed by OUTBOUND traffic to the Phone IP on ports 7076 and 7077. That will have to be passed at the Phone IP side.

    Welp, I set that field to my WAN IP and now it's working.  Thank you!