NAT Port Forwarding to Internal host UDP port 5060 not working as expected



  • Short Version:

    Pfsense 2.0 is NOT port forwarding UDP port 5060 to my Internal host (Unconditionally) like I'd expect.
    It's behaving like some kind of sick ALG and consistently dropping certain SIP ACK packets.. well not actually dropping them
    They pass on the wan rule (don't get logged as a drop) but they are not being natted and appearing on the LAN interface.

    More:

    I've been at this for a couple of weeks and now am at the point of pulling my hair out.

    I ever so badly want to use PFSense in our production environments but this is one stupid irritating problem that I have been unable to figure out
    has so far kept us using IPCOP and a number of other routing natting platforms that I long to move away from since I've seen PFSense.
    I don't mind paying for support if it can get this resolved or re-written to work.

    I've read many posts that suggest a number of settings to try setting up static outbound rules & the like.. however across trying all of these things that I
    have so far I am still seeing my issue where the SIP ack appears on the WAN interface but never makes it to the LAN interface as seen BOTH by doing packet captures
    from the PFSense box itself and looking at the incoming UDP port 5060 packets on the internal asterisk box.
    My problem is not with dynamic ports it with a simple port forward that is not working 'properly'.

    Pfsense is dropping or not natting these select packets (for reasons I have not been able to figure out) and also not logging anything about it.
    I did turn up the logging detail.

    As Captured on PFSENSE itself..  You can see that Packet 10 (see below) gets natted and shows up on the LAN interface But packet 11 never does.
    This is happening consistently and on all tested inbound SIP invites.. (We send out the INVITE but do not get the ACK all the way back to the endpoint at host:5060)
    Adjusting the rules & advanced settings over two weeks has not changed this behavior for me at all.
    I also reset state and or reboot every time I've tried something different just to be sure.

    (Hair pulling) I'm not sure if this is a bug or by design but it's not effing port forwarding UDP 5060 all the time like I expect.

    4.4.4.4 = my public IP
    5.5.5.5 = provider public IP
    10.73.73.8 = my internal asterisk host where ALL port forwarded UDP 5060 should arrive after I get it at the public Interface and it's natted.

    [WAN]

    No.     Time        Source                Destination           Protocol Length Info
        10 18.558549   4.4.4.4        5.5.5.5       SIP/SDP  1070   Status: 200 OK, with session description

    Frame 10: 1070 bytes on wire (8560 bits), 1070 bytes captured (8560 bits)
    Ethernet II, Src: RPTInter_30:50:8a (00:40:95:30:50:8a), Dst: SmcNetwo_63:03:fe (78💿8e:63:03:fe)
    Internet Protocol Version 4, Src: 4.4.4.4 (4.4.4.4), Dst: 5.5.5.5 (5.5.5.5)
    User Datagram Protocol, Src Port: 5060 (5060), Dst Port: sip (5060)
    Session Initiation Protocol

    No.     Time        Source                Destination           Protocol Length Info
        11 18.598757   5.5.5.5       4.4.4.4        SIP      547    Request: ACK sip:4150@4.4.4.4:5060

    Frame 11: 547 bytes on wire (4376 bits), 547 bytes captured (4376 bits)
    Ethernet II, Src: SmcNetwo_63:03:fe (78💿8e:63:03:fe), Dst: RPTInter_30:50:8a (00:40:95:30:50:8a)
    Internet Protocol Version 4, Src: 5.5.5.5 (5.5.5.5), Dst: 4.4.4.4 (4.4.4.4)
    User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
    Session Initiation Protocol

    [LAN]

    No.     Time        Source                Destination           Protocol Length Info
         10 18.558549    10.73.73.8            5.5.5.5       SIP/SDP  1073   Status: 200 OK, with session description

    Frame 9: 1073 bytes on wire (8584 bits), 1073 bytes captured (8584 bits)
    Ethernet II, Src: Vmware_c7:88:07 (00:0c:29:c7:88:07), Dst: 00:73:73:33:33:02 (00:73:73:33:33:02)
    Internet Protocol Version 4, Src: 10.73.73.8 (10.73.73.8), Dst: 5.5.5.5 (5.5.5.5)
    User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
    Session Initiation Protocol

    ACK to my 200 OK with SD should arrive here and it never does!

    Everything else I have done with PFSense Nat & Portforwarding has worked flawlessly and without a hitch until I ran into this problem.

    I'm very experienced with using Asterisk & SIP/RTP behind NAT and have made it work very well behind other Firewalls.
    Not bragging just wanting to save you some time in having to ask me if I've tried the very basics.
    Just have not been able to get this to work with my new favorite. :-(

    This is being tested in a VMWare Server environment and admittedly I need to try it out on bare metal with a couple of actual NICS but I expect to run into the same issue :-)
    We also really want this to work in said VMWare environment as well :-)
    There are number of posts out here where others have ran into this and essentially gave up and moved on to something else.
    I will continue to look and test.

    Please help or any thoughts!?

    THANKS!!!

    Steve



  • Do you have any packages installed on this pfsense like siproxy?

    What tool did you used to capture packates?

    Did you tried tcpdump via console?



  • Hello Marcelloc:

    Do you have any packages installed on this pfsense like siproxy?
    Absolutely NOT! :-)  I do not want this as I'd like to maintain a sip server behind PFSense the design of that
    Proxy is more for multiple sip endpoints behind a public IP. really the opposite of what I am doing.
    I also want ALL sip control and settings to be handled by me (non automatically).
    That proxy will tend to get in the way or cause me other problems/limitations.

    What tool did you used to capture packates?
    I used the 'packet capture' feature on the Web UI and made captures for both WAN & LAN.
    I also used Wireshark/Winpcap on separate computers via a monitor port setup on a managed switch.

    Did you tried tcpdump via console?
    I can, but have not as I have determined that PFSense is not natting all sip packets from traces listed above.

    Will running TCPDump at the prompt somehow give me any more information than doing a packet capture, saving the capture to a file
    and then opening it in wireshark like I have been doing?

    Also is there any way to get PFSense to log & display why these packets are not getting natted versus why all the other ones are?
    They certainly are not showing up in the firewall log as they fail to get forwarded.

    Thanks!!!  also thanks for responding so quickly!  :-)



  • I did not had this issues in a client using sip behind a 2.0 amd64 server.

    As you have voip skills, a way to workaround this nat problem until you find a solution for that is installing a sip server on firewall.

    You can install freeswitch from official repo or asterisk from my personal repo.

    freeswitch i386
    pkg_add -r http://files.pfsense.org/packages/8/All/freeswitch-1.0.6_1.tbz

    freeswitch amd64
    pkg_add -r http://files.pfsense.org/packages/amd64/8/All/freeswitch-1.0.6_1.tbz

    Asterisk i386
    http://e-sac.siteseguro.ws/packages/8/All/asterisk18-1.8.8.1.tbz

    Asterisk amd64
    pkg_add -r  http://e-sac.siteseguro.ws/packages/amd64/8/All/asterisk18-1.8.8.1.tbz

    I don't mind paying for support if it can get this resolved or re-written to work.

    Commercial support can be found here
    https://portal.pfsense.org/



  • Marcelloc, Thank you very much for responding and trying to help out.

    Installing PBX software on the same box to make it work is not an option for me.
    I need to get down to the bottom of why a simple port forward is not working and
    Forwarding ALL packets destined to UDP port 5060 to the Internal host like it's supposed to be doing.

    It shouldn't be dropping the packets like this period.  :-)

    I'll try this on separate AMD hardware with actual NIC hardware like you mention you have and see if the problem somehow goes away just by not doing it in a virtual machine
    test environment.

    You mention you don't have a problem using sip behind PFSense but you did not specify if you are talking about a separate asterisk server or just IP endpoints that register to an external
    SIP server..

    I do not have ANY problems when I test with just phones behind it (NAT) without any port forward rules that works just fine.

    I'm really doing the opposite of that.  Got an asterisk box behind it that needs to register to external places as well as allow static inbound on UDP port 5060 to allow others to register to me.

    It all works perfect on every other firewall I use and test except PFSense. :-(



  • 10 18.558549  4.4.4.4        5.5.5.5      SIP/SDP  1070  Status: 200 OK, with session description

    What happens if you make Source = ANY?



  • I'm not sure exactly what you are asking because when defining a NAT port forward Source = ANY by DEFAULT.



  • I just completely and entirely re-did my setup without anyvirtual machines and using actual hardware servers with actual nics and am having the exact same problem.
    I'm pretty convinced at this point that this is a serious bug and I'll need to report it.

    I'm asking the box to simply port forward UDP 5060  and it should port forward every single UDP DST port 5060 packet period and it is not.
    It's appearing to be doing some kind of sip or otherwise inspection and not passing or natting the packet to the lan interface and it should absolutely NOT be working this way.
    It's also not logging any dropped packets or reasons for not forwarding the packets as it occurs.
    The inbound packet looks fine and is addressed to the WAN address with a destination UDP port 5060.
    All other packets are making it across and appearing natted onto the LAN but these particular SIP ack packets are not.

    If I tell it to portforward UDP 5060 from anywhere to the LAN destination 10.73.73.8 UDP 5060 it should do this and it is not.
    This is VERY broken. and is stopping me from being able to use the asterisk box with PFSense which is incredibly frustrating right now.

    Thank you again very much for trying to help.

    :-)



  • I think what I'm running into here is that PFSense is trying to be WAY too application or WAY too SIP aware and making choices not to route/nat inbound based on various crap.
    I'm now running into a similar problem with inbound RTP packets..

    They're not getting natted!! (and it seems to be due to PFSense randomizing outbound source ports) which I should be able to deal with with static outbound settings).

    I really should be able to for at least troubleshooting purposes be able to tell the firewall to blindly accept all UDP packets in a port range and just NAT them and forward them on to the LAN!
    But it's making stupid decisions NOT to do this based on the state of OTHER connections to the same port or port ranges!!

    I can recreate the UDP port 5060 problem with ease..
    It's happening when the SIP trunk provider sends me an ACK packet on port 5060 after I have registered from a random source port..
    Firewall is determining that this packet is somehow not associated or 'off color' because it is not destined to my source port..
    This is bullshit because my port forwarding rule should override this!! and I should be getting the packet passed on to my internal host.
    PFSense is incorrectly acssociating this packet to the fact that it came from the same host I'm talking to sending me packets to port 32701
    And ignoring /dropping any new packets coming from the same distant host that are not associated with the first SIP connection.
    This is INSANE!  (Or I am) :-)

    There really really REALLY should be a way to bypass this behaviour and portforward inbound packets to the right place unconditionally.

    I'm starting to see a pattern or mentality where the dynamically created reachback rules are somehow overriding the static port forward rules in some cases
    which is completey the opposite of how it should work.. a port forwarding put in place should ALWAYS work regardless of what's going on dynamically.



  • Alright I seem to have it all working now.. just a LOT of working through it and pissing & moaning as I went along.

    If I setup NAT static outbound for the host to force only outbound SRC port 5060 my inbound 5060 packets now work coming back but ONLY because of the dynamic pipe created by forcing it to use port 5060!
    This is a HIDEOUS ugly and painful port forwarding bug a half which I shall report! :-)

    Obviously if I had a sip registration with a SRC port of 34211 and the ITSP goes and sends me their ACKS to port 5060 but everything else to port 34211 (this is something they should not be doing)
    Of course with sip that's debatable.. it may not actually be improper but it's weird.
    But if portworwarding were actually working properly (which it's not) I'd get that improper or unexpected port 5060 SIP ACK packet to the asterisk box (as I do with any other firewall doing port forwarding) and all would be happy & work.



  • @N8LBV:

    I think what I'm running into here is that PFSense is trying to be WAY too application or WAY too SIP aware and making choices not to route/nat inbound based on various crap.

    Not true, UDP is UDP, SIP isn't handled any differently from any other UDP traffic, there is 0 SIP intelligence in the underlying firewall.



  • [WAN]

    No.    Time        Source                Destination          Protocol Length Info
        10 18.558549  4.4.4.4        5.5.5.5      SIP/SDP  1070  Status: 200 OK, with session description

    Your rule example above seems to indicate you are restricting incoming from a single IP…

    Under source-  4.4.4.4, make any...  But may be moot now.

    pfSense by default obfuscates the outgoing port.  From what I know SIP client devices expect response from the same port that they sent to...  IE  5060, 5080, (vonage-) 10000...

    If your running a sip server you should have static port on outgoing connections from that server.

    If you can use 1:1 NAT.



  • Not true, UDP is UDP, SIP isn't handled any differently from any other UDP traffic, there is 0 SIP intelligence in the underlying firewall.

    Well Good!! That's the way it should be however I'm seeing that my configured rule is NOT being adhered to
    I have a port forward rule that specifies to allow from any as suggested ans is obvious (default) by the way.
    And not all packets are being sent on ..
    My observation is that it appears to be conditional as to what previous dynamic connections have been
    made to the host that I am not getting the packets natted to the internal host from..

    This: "

    [WAN]

    No.    Time        Source                Destination          Protocol Length Info
        10 18.558549  4.4.4.4        5.5.5.5      SIP/SDP  1070  Status: 200 OK, with session description

    IS NOT A RULE!  it is just the packet I am recieving at the WAN that should be getting natted and passed to the LAN but is not. It's a Text export from wireshark.



  • This is still looking like a very serious bug.
    I'm working on a test scenario where this can be duplicated and recreated in the lab
    easily with hooking up a few machines or doing it all within a virtual environment between a couple of virtual machines simulating the scenario.
    And I'll pass my test setup over to the developers so they can easily recreate and replicate + see the problem I am talking about.

    The issue is quite clear.. with a port forward rule specifying that all UDP from anywhere with a DST port of 5060 should come in be allowed natted and passed to a specific internal host, PFSense should simply do just that and it is not.

    Depending on other (dynamic) connections inbound on other ports from the same external host that is sending me packets on UDP 5060 (Which I'm getting on the WAN) I'm seeing PFSense fail to nat and send these specific inbound packets to the LAN interface and as a result they never reach the internal host.



  • If I have time this weekend, I'll try to reproduce this.

    It happens will all 5060 udp packages or some packages?

    Just to be 100% sure we have same rdr nat

    On Firewall -> nat -> Port Forward




  • It only happens with some specific packages.
    If I have a dynamic SIP registration going out –>10.73.73.8:(src)5060:(dst)5060-->PFSMyPublicIP:43112:5060-->RemotePublic IP:43112:5060
    Packets coming back as RemotePublicIP:5060:43112--->PFSMyPublicIP:43112:5060--->10.73.73.8:5060:5060 work good as expected.

    However if RemotePublicIP suddenly sends us another packet like this:  RemotePublicIP:5060:5060 (instead or aside from) RemotePublicIP:5060:43112
    The packet destined to port 5060 does not get nated and send on to 10.73.73.8 like it should be due to the port forwarding rule.
    It's as if the dynamically created reachback rule is somehow overwriting the port forward rule which should NEVER happen :-)

    My test scenario is this:

    From PFSense Fresh install Defaults

    1. assign single static IP to WAN Interface.
    2. assign single static IP to lan interface. (10.73.73.1)
    3. assign default gateway and DNS address for WAN
    4. DHCP disabled on lan & wan
    5. Start with auto outbound rule make sure Internet access & nat Lan to outside is working
    6. Enable NOT blocking bogon networks (not needed to recreate problem) I just do it for testing.
    7. Internal asterisk box placed at 10.73.73.8 and using PFS as it's gateway.
    8. Asterisk nat settings are set properly for internal network range and external public IP address in sip.conf
    9  In PFS Set Port forward from anywhere UDP DST port=5060 gets rdirect target set to 10.73.73.8 redirect port UDP 5060
    10. In PFS Set Port forward from anywhere UDP DST Ports =11200-11300 redirect to target 10.73.73.8 UDP DST 11200-11300 (this is for RTP and NOT needed to demonstrate port 5060 failure to nat).
    In my case port forward 5060 fail issue can still be tested without without RTP flowing to the internal box You just simply won't have working audio has nothing to do with the issue reported.

    11. Have asterisk register to an external SIP provider will by default create a dynamic inbound port back to asterisk (THIS IS OK).
    12. You are now registered with provider outbound port is UDP 5060 inbound port will be random 32119 for example.. Inbound SIP from registered provider will be on UDP port 32119.
        (my provder doesn't care if we randomize or dynamically change the outbound port it still works)
    13. Same provider now sends you usual expected packets on port 32119 for SIP messaging as expected and they reach the internal box just fine.
    14. TEST: Now same provder sends us another SIP packet but instead they send it to UDP PORT 5060 instead of 32119.
    15.  PFSense gets the packet but does not nat it and does not pass it on to 10.73.73.8 regardless of our port forward rule that it should.

    I can provide more details and packet captures and screenshots of my PFSense setup if this will save you any time as well as allow you remote access to my test config via Private message email or a phone call.

    Also just confirming the focus here is now that I'm siting a bug in port forwarding.
    I did get my system to 'work' by Forcing static outbound on UDP 5060 just to get PFSense get me the packets coming back on PORT 5060 via a separate mechanism other than a port forward rule.
    aside from port forwarding rules.. This made my system work but also setup other ugly limitations that will cause me grief later on for sure when I need other (random ports) to work in combination with
    5060 to/from the same remote host.



  • You need to rewrite the source port on your outbound SIP as is done by default. I suspect the outbound NAT state is picking up the traffic rather than the rdr (port forward), because you aren't rewriting the source port.



  • You need to rewrite the source port on your outbound SIP as is done by default.
    Sorry I'm not understanding exactly what you are saying here.
    If I use it in a default fashion while outbound sip is being rewritten I am seeing the problem with inbound packets to port 5060 which is wrong.
    A remote host can send inbound packets to multiple ports there is nothing wrong with this.

    I suspect the outbound NAT state is picking up the traffic rather than the rdr (port forward), because you aren't rewriting the source port.
    Still not understanding at all (sorry). And we are dealing with an inbound static rule so outbound nat state should not matter.

    I still have an expectation as it works on all other firewalls and that is if you define a port forward rule it will work regardless of what is going on
    dynamically.

    I still see no reason at all why this is not portwarding under all conditions. That is the sole purpose of port forwarding. Seriously.. there's no reason for it NOT to work unless
    there is a bug in the packet inspection and/or decision making here.. It's a brain dead simple port forwarding task and it's failing.



  • CMB, it sounds like you are telling me what I need to do to make it work (which I have done) and have made it work awhile back.. maybe you already read this.. I'm not sure.  I have done exactly that and it does work. But portworwarding itself in this case still does not behave the way I expect it should.
    It breaks other things that I won't dig into right now unless you ask, as it will take this off topic.
    I'll continue to listen & learn for awhile before I open a bug report in case I'm missing something.  Thank you
    so very much for helping out!



  • I am having a similar problem but I can at least get it to work using Manual NAT. The problem with this is once I turn on manual NAT with Static, it breaks my other subnets web browsing capabilities.

    Here is my scenario:

    192.168.0.0 - Local lan
    192.168.1.0 - VPN lan
    192.168.12.0 - MPLS lan
    192.168.13.0 - MPLS lan
    192.168.14.0 - VPN lan
    192.168.23.0 - VPN Lan
    192.168.24.0 - VPN lan
    192.168.25.0 - VPN lan
    192.168.26.0 - VPN lan

    The phone system sits on 192.168.0.6 and there are IP phones on 192.168.23.0 and 192.168.25.0 working fine. We had to add a SIP provider for routing "out of area" numbers to the phone system from the phones sitting on the  192.168.14.0 network. The intercom traffic works fine between 192.168.0.0 and 192.168.14.0 through the VPN. All HTTP traffic works fine through squid3 as a proxy from all networks. With this currently set as Auto Outbound NAT everything works fine except the SIP traffic. I can call the number, the packets get routed through the phone system and the phones at 192.168.14.0 ring. I can hear them answer but they cannot hear me. To resolve this I turned on Manual Outbound NAT. Fixed the phone issue, but now no one can access web pages through squid3 on any network other than 192.168.0.0 which is local to the firewall and the phone system.

    I have come to the conclusion that the Static Port option breaks squid3 but fixes the phones, and in turn static mapping turned off fixes squid3 but breaks the phones (one way audio). We are using an outside provider so there are no SIP equipment onsite.

    Any one have any suggestions or need further info please feel free. I'm at the point I'll try anything now.



  • i am having the exact same problem as N8LBV.  i have been using voip through pfsense for years with no problems until setting up a new box with version 2.0.1 of pfsense.  something isnt right.  i have tried all the suggestions for outbound static ports and anything else that people suggest in various forums and still no love.  why can't this version of pfsense just not forward udp traffic that originated from behind the firewall back to it on port 5060 to my voip server?  i can connect to my server through port 5060 on a sip client that originates on the wan connection, but not a connection that originated from behind pfsense, i hope that makes sense.



  • I finally got mine working today. i reset everything to factory defaults.  created a nat port forward
    WAN TCP/UDP * * WAN address 5060 (SIP) 10.0.0.8 5060 (SIP) and allowed it to create a new associated filter rule for that.  under NAT.outbound i changed it to manual and set it to use static-port, reloaded my trunk and inbound calls worked.



  • hi all,

    i also have the exact same issue – 9 packets arriving on wan, 2 arriving at interior host.

    i'm running a pfsense 2.01 host on amd64 inside a vmware virtual machine.

    seriously grave -- i cannot seem to get the packets forwarded following the advice in this thread :-(

    c



  • Same issue here. Unfortunately the tips above didn't worked for me :( Any suggestion ?



  • it seems to work on 2.1 snapshot of july 18th



  • N8LBV,

    does the SIP UA on the inside of your pfSense implement rport? If not, I would expect that SIP responses coming back from the SIP server on the outside will be sent to the port specified in the via header inserted by the SIP UA on the inside of your network, which for SIP over UDP normally would be 5060.

    Can you post a packet capture showing an outbound SIP request and the corresponding response which is coming back on the wrong port?



  • N8LBV,

    also keep in mind that while SIP responses use via headers for making it back to the originator, new requests within the same transaction (Such as the ACK request you mentioned in your initial post) use record-route headers for making it to their destination, which could explain why some SIP responses/requests are not arriving at the same port as the outbound traffic used.



  • I am also having the exact same problem. A brain dead port forward on UDP port 5060 is not working. I have been pulling my hair out trying everything. Packet captures on the pfsense show that the retuning packets never get forwarded from with WAN to LAN, i.e. are on WAN but never on LAN. I reset to defaults and added just the simple rule and still no joy.



  • Gofast,

    have you verified that the response is coming back to the same port as the original request was sent out from?

    If the involved SIP UA's do not support the 'rport' SIP extension then the responses would be sent back to the port specified in the topmost via header, which would normally not be the same port as the NAT'ed port the original request was sent out with.

    • Andreas


  • The source and dest port were 5060 from the asterisk box for the packet on the LAN being sent to the internet(WAN).
    The source port got changed via the outgoing NAT to the WAN. The return packet from the internet had source and dest ports of 5060. The port forward should forward this regardless because it met the rule - destination port 5060 on wan forward to an ip on the LAN. I firewall added rules to allow all on all interfaces and it doesn't help. It appears to just get dropped for an unknown reason (unknown to me). Why does the port forward get ignored? What came before shouldn't matter as UDP is stateless. Am I missing something?



  • Gofast,

    can you post screenshots of the NAT PFW rule and the associated firewall rule?



  • I can post tomorrow as I don't have access right now. The 2.01 seem to have a bug that changing the fwd rule from tcp to udp didn't update the auto created filter to udp (and you couldn't edit it). This didn't seem to occur in 2.1. Ether way with the rules+filter created correctly I couldn't make it work.

    It was created with the following settings
    protocol udp
    destination port 5060 - 5060
    target ip - lan ip of voip box i.e. 10.x.x.x
    target port 5060
    everything else was default i.e. Interface - wan, destination - wan address, create associated rule, no source options selected.

    Diagnostics / packet capture showed packets sending & receiving on wan but only packets destined for internet on lan. tcpdump on voip box also confirmed this. The wan is a pppoe to modem in bridge mode.

    I also tried manual outgoing nat, adding rues to allow everything on all interfaces, advanced options - Bypass firewall rules for traffic, Disable the PF scrubbing option,  Reflection on/off



  • I can't get port forwards on UDP port 5060. I can get a mail server, pptp server, terminal server, web server all working (TCP and TCP + GRE for PPTP), but not VOIP on UDP.
    Here are the packet captures from the interfaces.

    10.X.X.X is my VOIP box address on the LAN
    122.X.X.X is the WAN address
    213.X.X.X is the VOIP providers address

    Packet captures on the voip box on the LAN - I can only see packets being sent to the VOIP provider…
    20:00:37.425557 IP 10.X.X.X.5060 > 213.X.X.X.5060: SIP, length: 648
    20:00:38.425626 IP 10.X.X.X.5060 > 213.X.X.X.5060: SIP, length: 648
    20:00:40.425756 IP 10.X.X.X.5060 > 213.X.X.X.5060: SIP, length: 648
    20:00:44.426029 IP 10.X.X.X.5060 > 213.X.X.X.5060: SIP, length: 648

    packet captures on the pfsense on the WAN if - I can see packets being sent to the VOIP provider...and responses back.
    20:22:14.587253 IP 122.X.X.X.56607 > 213.X.X.X.5060: UDP, length 514
    20:22:14.634733 IP 213.X.X.X.5060 > 122.X.X.X.5060: UDP, length 455
    20:22:14.816183 IP 122.X.X.X.56607 > 213.X.X.X.5060: UDP, length 514
    20:22:14.863381 IP 213.X.X.X.5060 > 122.X.X.X.5060: UDP, length 455

    packet captures on the pfsense on the LAN again only show packets being sent to the VOIP provider.
    20:24:21.581862 IP 10.X.X.X.5060 > 213.X.X.X.5060: UDP, length 514
    20:24:21.808852 IP 10.X.X.X.5060 > 213.X.X.X.5060: UDP, length 514
    20:24:24.543081 IP 10.X.X.X.5060 > 213.X.X.X.5060: UDP, length 648
    20:24:28.543369 IP 10.X.X.X.5060 > 213.X.X.X.5060: UDP, length 648

    I've attached a screenshots of the NAT rule. I've used the auto created firewall rule. I've tried the latest build, allowing everything from anywhere to anywhere of any type on everything, manual nat with static port, nothing helps. Nothing in the logs either. The responses appear to be just dropped for no apparent reason. Anyone have any idea what I should try next? Oh and it works on my Cisco 877.




  • gofast, perhaps you could try changing the NAT rule to have a 'Dest.addr' of 'WAN address' instead of the manually input 122.x.x.x IP address?

    If you have a static WAN IP it shouldn't make any difference I guess, but if your WAN is using DHCP, it would be safer to have 'Dest.addr' set to 'WAN address' since that alias would always be your current WAN IP address.



  • Hi,

    It is a static ip address so yes it shouldn't make any difference. The wan connection is pppoe to a modem in bridge mode. Anyway you can see from the packet capture that the packets are arriving back with the correct address. All other tcp connections nat just fine. How do I log a bug for this? I'm thinking of setting up a virtual environment to replicate the problem for the developers. What vitalisation technology should I use for this? Virtual box? I suspect this is the same issue I get with pptp connections sometimes not working. Perhaps the same problem with udp packets is affecting gre packets? (also stateless packets). Also how can I access the full nat table (not just the summary)?
    I've spent many hours on this and am sure it is a bug. I've reset to defaults, upgraded to latest release, etc. etc. etc. If the rule says nat the packet and it matches the rule and is not being filtered, Why isn't it working  ???

    Regards,
    Tony



  • Hey guys saw this is a recent thread and I am having the exact same issue. I tried upgrading to 2.1 but still having the issue. I've tried port forwarding, ive tried static outgoing nat, set the firewall to conservative, still no dice. I am having an issue with the FreeSWITCH pbx behind the firewall where incoming calls drop after exactly 32 seconds. Doing packet caps, I traced it to the firewall not forwarding SIP 'ACK' packets along to the pbx. I've attached the flow of 3 packet caps - one on the pfsense lan, one on wan, and one on the pbx. I have phones registered directly to the ip trunk carrier now as a workaround since my pbx is not fully functional due to this issue. In the packet caps the phones all ring for about a half second then the PBX auto-answers and routes to the IVR menu so the phones stop ringing after half a ring. You can see in the packet caps that the phones receive the incoming ACK packets fine, and note that the phones are using random ports, while the PBX is always using 5080. Maybe this has something to do with it?
    Please note that the LAN capture and WAN capture were done for different calls, as I did not know how to do both at once with pfSense. The PBX capture was done at the same time as the pfSense WAN capture IIRC.

    Edit: Removed the images as they are not really relevant to this thread. We somewhat solved our issue as you can see below, but I plan to do some more testing with the static nat options and such to see if my install exhibits the same behavior as the OP's.



  • Just to let everyone know, we resolved our issue. Here is a summary of what we are working with:
    pfSense 2.1 box on a Verizon FIOS connection
    FreeSWITCH PBX on internal LAN, using SIP registration. All endpoints (phones) are on the internal lan as well.

    To fix this issue, we needed to modify the way our PBX behind the NAT formed its SIP packets. It was formatting the Contact field in the SIP headers with the external WAN IP and the PBX external port. When the UDP packets passed out from the pbx and through pfSense, pfSense was randomizing the source port. The voice carrier's systems seem to work in a way where if the IP in the contact field matches the source IP of the packet, it chooses to use that return IP along with the port shown in the contact field. Since the port was set as the PBX's external port, and pfSense was randomizing the port, the return traffic from the voice provider was addressed to that pbx port, not the pfsense port, and the traffic was dropped. When we modified the PBX configuration to use its local LAN ip in the contact field, the voice provider's systems chose to disregard the contact field, and would return the udp packets to the source port that pfSense set, and everything would work ok.

    So I will try to document how this stuff works so anyone out there struggling with the same issue can understand how the symmetric NAT works with SIP and VOIP. Now there are two main elements to VOIP:
    SIP Signalling (controls the flow of the call, can contain DTMF signalling and controls the beginning, ending, routing etc of the call)
    RTP Voice Stream (voice and possibly video stream, also contains DTMF signalling in some cases - inband or oob)

    The signalling is a separate channel from the rtp data. The rtp data is always UDP. The SIP signalling can be either UDP OR TCP, depending on how your solution is configured.

    RFC 4961, section 4 states:

    There are two specific instances where symmetric RTP and symmetric
      RTCP are REQUIRED:

    The first instance is NATs that lack integrated Application Layer
      Gateway (ALG) functionality.  Such NATs require that endpoints use
      symmetric UDP ports to establish bidirectional traffic.  This
      requirement exists for all types of NATs described in Section 4 of
      [RFC4787].  ALGs are defined in Section 4.4 of [RFC3022].

    Other UDP-based protocols can also benefit from common local transmit
      and receive ports.

    There are no known cases where symmetric RTP or symmetric RTCP are
      harmful.

    For these reasons, it is RECOMMENDED that symmetric RTP and symmetric
      RTCP always be used for bidirectional RTP media streams.

    I have observed pfSense using the same port internally as it does externally for RTP streams. However, it does not use the same port as the remote server does.

    As for SIP signalling over TCP and UDP, so far I have used purely UDP, although I plan to experiment with TCP. With non-rtp UDP traffic, I don't believe RFC says anything about ports needing to stay the same. In my experience, pfSense uses a random external port with non-rtp udp traffic. I believe this is normal functionality. According to a tech at our voice provider, with SIP signalling the ports on both ends SHOULD BE be the same for proper functionality. I am not sure whether the issue we had is common to all ip voip trunking providers, I would have to see if there is a spec saying anything about the contact field in the sip header. But it seems we worked around the issue of ports not being the same by setting an internal ip for the contact field so that the remote server uses the originating ip/port to return signalling traffic to. So from what I understand, it is good practice to keep the port that the PBX behind the firewall is using for signalling exactly the same on the outside of the firewall. I believe this is known as 'port preservation' in a NAT system.

    Now for TCP SIP signalling, I have not had a chance to experiment, but I am assuming that the same functionality will be exhibited with a random port being used on the WAN interface.

    Now I am going to attempt to figure out the best way to keep the ports symmetric in pfSense and I will post back. The outgoing static nat settings in pfSense may be the answer, but when I previously tested it, the ports seemed to still be asymmetric. But I now realize that I may need to clear the state table before the change will take effect, so I will try that next.

    To work with a NAT device, a voip device on the LAN needs to either:
    A. Set the SIP header contact field to its LOCAL LAN IP & PORT
    B. Set the SIP header contact field to the EXTERNAL WAN IP & PORT

    Now there is a big caveat with B - the VOIP device MUST KNOW ITS EXTERNAL PORT. With a symmetric nat, this becomes tricky as the external port will be randomized. In this case, it is best to set the contact field to the internal lan ip and port, because some voip providers will mistakenly try to use the info from the contact field if the external IP matches but the port is not correct because of symmetric nat. In my case, I had the voip device set to use auto-nat. Some devices also use STUN servers to discover their external ip. It seems that these methods discovered the external IP, but did NOT discover the external port properly or at all (it was still using the internal port in the contact field). Even if the external server did discover the external port, the symmetric nat would use a different random port for two different connections so it may not work properly.

    Seems like the moral of the story is that VOIP is pretty tricky sometimes.
    Anyone else that has info, please chime in. I am not an expert, still learning this stuff.



  • @launch3:

    Now this is all fine and dandy, but in the case of pfSense, some clearing of the air is needed. I read somewhere (unsure of the accuracy of this statement), that pfSense did NOT contain any ALG functionality. However, according to a tech at the voice provider, with SIP the ports on both ends MUST be the same.

    ...

    This leads me to believe that pfSense does indeed have ALG functionality.

    I'd like to add a few comments here. What you are describing here, where pfSense randomizes the source port on outbound traffic, is not ALG functionality. ALG, in this case SIP ALG, means manipulating the SIP payload. That would for example mean manipulating the IP address and/port number in the contact header, or manipulating IP addresses and/or port numbers within the SDP of a SIP INVITE request.

    You describe that your SIP provider will send return traffic to the port specified in the contact header if the IP address in the contact header matches the IP address for which the SIP request was received from. This is, as far as my knowledge goes, not a behavior which is specified in RFC3261 or other SIP standards, but rather a "proprietary" behavior.

    From what you describe about how you managed to get this working, it may seem as your SIP provider supports rport (http://www.ietf.org/rfc/rfc3581.txt).

    I believe that if you use manual outbound NAT in pfSense, so that pfSense does not randomize the outgoing source port, but rather uses a port specified by you (Normally 5060), that this would also work since the response from your SIP provider would then come back to a port which you have pre-defined.

    Also, when you are quoting RFC 4961, that applies to RTP and RTCP, which is the actual audio/video media packets, which is not related to SIP signaling itself, which was the issue here.

    Just my $0.02.

    • Andreas


  • @inflamer:

    @launch3:

    Now this is all fine and dandy, but in the case of pfSense, some clearing of the air is needed. I read somewhere (unsure of the accuracy of this statement), that pfSense did NOT contain any ALG functionality. However, according to a tech at the voice provider, with SIP the ports on both ends MUST be the same.

    ...

    This leads me to believe that pfSense does indeed have ALG functionality.

    I'd like to add a few comments here. What you are describing here, where pfSense randomizes the source port on outbound traffic, is not ALG functionality. ALG, in this case SIP ALG, means manipulating the SIP payload. That would for example mean manipulating the IP address and/port number in the contact header, or manipulating IP addresses and/or port numbers within the SDP of a SIP INVITE request.

    You describe that your SIP provider will send return traffic to the port specified in the contact header if the IP address in the contact header matches the IP address for which the SIP request was received from. This is, as far as my knowledge goes, not a behavior which is specified in RFC3261 or other SIP standards, but rather a "proprietary" behavior.

    From what you describe about how you managed to get this working, it may seem as your SIP provider supports rport (http://www.ietf.org/rfc/rfc3581.txt).

    I believe that if you use manual outbound NAT in pfSense, so that pfSense does not randomize the outgoing source port, but rather uses a port specified by you (Normally 5060), that this would also work since the response from your SIP provider would then come back to a port which you have pre-defined.

    Also, when you are quoting RFC 4961, that applies to RTP and RTCP, which is the actual audio/video media packets, which is not related to SIP signaling itself, which was the issue here.

    Just my $0.02.

    • Andreas

    Andreas, thanks for your input. Regarding the misinterpretation of the RFC, I realized my mistake and was in the middle of editing my post when you posted. Please see my revised post. As for the manual outbound NAT, my testing did not work, BUT i feel that may be because the state table needed to be cleared after that change. It seemed that the port that was set in the state table was still being used the whole time. I also checked out the spec for RPORT that you linked - there is no RPORT appearing in the sip headers that I captured.

    Here someone basically sums up what I have experienced:
    http://qutecom.org/ticket/27

    It's easy: some providers (as Ekiga) don't allow private address in Contact, this means that the provider doesn't help when the client is behind NAT. This meants that the provider doesn't fix the signalling (by using received public address instead of private "Contact") but that the provider doesn't offer a RTP tunnel (proxy RTP).
    In this case the only solution is in the client side which could be:
    a) STUN support: If the client is behind a non symmetric NAT router and supports STUN, it can solve by itlsef the signalling and media (by setting public addresses in "Contact" and SDP (retrieved using STUN).
    b) Manual port forwarding in the router and setting these values manually in the signalling and SDP sent by the client.

    In my case, setting the contact field to the private address disqualified it in the providers system, whereas with a publically routable address, the provider attemped to route the return packets to it. It may not be 'if the ip in the contact field matches the source ip' as I stated earlier, and they may just be looking for a private or public ip in there, since IIRC there are certain ip ranges that are always reserved for private network addressing.

    I am realizing more and more that this is not really pfSense specific, it's more of a snafu when dealing with SIP and symmetric NAT.



  • Hi All,

    I'm definitely getting replies from my voip service provider (type udp, dest =  my wan ip:5060) they just aren't getting forwarded by my simple port forward rule (posted previously). I can't see a reason why the pfSense is dropping them. The packet trace (also posted previously) clearly shows the arriving packets on wan but nothing forwarded to lan. Can someone tell me what possible reasons a port forward wouldn't work? I've put firewall rules in place to allow everything on every interface. It looks to me like the forward rule is being ignored for some reason but the packet matches it.

    Regards,
    Tony


Log in to reply