NAT Port Forwarding to Internal host UDP port 5060 not working as expected
-
Short Version:
Pfsense 2.0 is NOT port forwarding UDP port 5060 to my Internal host (Unconditionally) like I'd expect.
It's behaving like some kind of sick ALG and consistently dropping certain SIP ACK packets.. well not actually dropping them
They pass on the wan rule (don't get logged as a drop) but they are not being natted and appearing on the LAN interface.
–
More:I've been at this for a couple of weeks and now am at the point of pulling my hair out.
I ever so badly want to use PFSense in our production environments but this is one stupid irritating problem that I have been unable to figure out
has so far kept us using IPCOP and a number of other routing natting platforms that I long to move away from since I've seen PFSense.
I don't mind paying for support if it can get this resolved or re-written to work.I've read many posts that suggest a number of settings to try setting up static outbound rules & the like.. however across trying all of these things that I
have so far I am still seeing my issue where the SIP ack appears on the WAN interface but never makes it to the LAN interface as seen BOTH by doing packet captures
from the PFSense box itself and looking at the incoming UDP port 5060 packets on the internal asterisk box.
My problem is not with dynamic ports it with a simple port forward that is not working 'properly'.Pfsense is dropping or not natting these select packets (for reasons I have not been able to figure out) and also not logging anything about it.
I did turn up the logging detail.As Captured on PFSENSE itself.. You can see that Packet 10 (see below) gets natted and shows up on the LAN interface But packet 11 never does.
This is happening consistently and on all tested inbound SIP invites.. (We send out the INVITE but do not get the ACK all the way back to the endpoint at host:5060)
Adjusting the rules & advanced settings over two weeks has not changed this behavior for me at all.
I also reset state and or reboot every time I've tried something different just to be sure.(Hair pulling) I'm not sure if this is a bug or by design but it's not effing port forwarding UDP 5060 all the time like I expect.
4.4.4.4 = my public IP
5.5.5.5 = provider public IP
10.73.73.8 = my internal asterisk host where ALL port forwarded UDP 5060 should arrive after I get it at the public Interface and it's natted.[WAN]
No. Time Source Destination Protocol Length Info
10 18.558549 4.4.4.4 5.5.5.5 SIP/SDP 1070 Status: 200 OK, with session descriptionFrame 10: 1070 bytes on wire (8560 bits), 1070 bytes captured (8560 bits)
Ethernet II, Src: RPTInter_30:50:8a (00:40:95:30:50:8a), Dst: SmcNetwo_63:03:fe (78
8e:63:03:fe)
Internet Protocol Version 4, Src: 4.4.4.4 (4.4.4.4), Dst: 5.5.5.5 (5.5.5.5)
User Datagram Protocol, Src Port: 5060 (5060), Dst Port: sip (5060)
Session Initiation ProtocolNo. Time Source Destination Protocol Length Info
11 18.598757 5.5.5.5 4.4.4.4 SIP 547 Request: ACK sip:4150@4.4.4.4:5060Frame 11: 547 bytes on wire (4376 bits), 547 bytes captured (4376 bits)
Ethernet II, Src: SmcNetwo_63:03:fe (788e:63:03:fe), Dst: RPTInter_30:50:8a (00:40:95:30:50:8a)
Internet Protocol Version 4, Src: 5.5.5.5 (5.5.5.5), Dst: 4.4.4.4 (4.4.4.4)
User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
Session Initiation Protocol[LAN]
No. Time Source Destination Protocol Length Info
10 18.558549 10.73.73.8 5.5.5.5 SIP/SDP 1073 Status: 200 OK, with session descriptionFrame 9: 1073 bytes on wire (8584 bits), 1073 bytes captured (8584 bits)
Ethernet II, Src: Vmware_c7:88:07 (00:0c:29:c7:88:07), Dst: 00:73:73:33:33:02 (00:73:73:33:33:02)
Internet Protocol Version 4, Src: 10.73.73.8 (10.73.73.8), Dst: 5.5.5.5 (5.5.5.5)
User Datagram Protocol, Src Port: sip (5060), Dst Port: sip (5060)
Session Initiation ProtocolACK to my 200 OK with SD should arrive here and it never does!
Everything else I have done with PFSense Nat & Portforwarding has worked flawlessly and without a hitch until I ran into this problem.
I'm very experienced with using Asterisk & SIP/RTP behind NAT and have made it work very well behind other Firewalls.
Not bragging just wanting to save you some time in having to ask me if I've tried the very basics.
Just have not been able to get this to work with my new favorite. :-(This is being tested in a VMWare Server environment and admittedly I need to try it out on bare metal with a couple of actual NICS but I expect to run into the same issue :-)
We also really want this to work in said VMWare environment as well :-)
There are number of posts out here where others have ran into this and essentially gave up and moved on to something else.
I will continue to look and test.Please help or any thoughts!?
THANKS!!!
Steve
-
Do you have any packages installed on this pfsense like siproxy?
What tool did you used to capture packates?
Did you tried tcpdump via console?
-
Hello Marcelloc:
Do you have any packages installed on this pfsense like siproxy?
Absolutely NOT! :-) I do not want this as I'd like to maintain a sip server behind PFSense the design of that
Proxy is more for multiple sip endpoints behind a public IP. really the opposite of what I am doing.
I also want ALL sip control and settings to be handled by me (non automatically).
That proxy will tend to get in the way or cause me other problems/limitations.What tool did you used to capture packates?
I used the 'packet capture' feature on the Web UI and made captures for both WAN & LAN.
I also used Wireshark/Winpcap on separate computers via a monitor port setup on a managed switch.Did you tried tcpdump via console?
I can, but have not as I have determined that PFSense is not natting all sip packets from traces listed above.Will running TCPDump at the prompt somehow give me any more information than doing a packet capture, saving the capture to a file
and then opening it in wireshark like I have been doing?Also is there any way to get PFSense to log & display why these packets are not getting natted versus why all the other ones are?
They certainly are not showing up in the firewall log as they fail to get forwarded.Thanks!!! also thanks for responding so quickly! :-)
-
I did not had this issues in a client using sip behind a 2.0 amd64 server.
As you have voip skills, a way to workaround this nat problem until you find a solution for that is installing a sip server on firewall.
You can install freeswitch from official repo or asterisk from my personal repo.
freeswitch i386
pkg_add -r http://files.pfsense.org/packages/8/All/freeswitch-1.0.6_1.tbzfreeswitch amd64
pkg_add -r http://files.pfsense.org/packages/amd64/8/All/freeswitch-1.0.6_1.tbzAsterisk i386
http://e-sac.siteseguro.ws/packages/8/All/asterisk18-1.8.8.1.tbzAsterisk amd64
pkg_add -r http://e-sac.siteseguro.ws/packages/amd64/8/All/asterisk18-1.8.8.1.tbzI don't mind paying for support if it can get this resolved or re-written to work.
Commercial support can be found here
https://portal.pfsense.org/ -
Marcelloc, Thank you very much for responding and trying to help out.
Installing PBX software on the same box to make it work is not an option for me.
I need to get down to the bottom of why a simple port forward is not working and
Forwarding ALL packets destined to UDP port 5060 to the Internal host like it's supposed to be doing.It shouldn't be dropping the packets like this period. :-)
I'll try this on separate AMD hardware with actual NIC hardware like you mention you have and see if the problem somehow goes away just by not doing it in a virtual machine
test environment.You mention you don't have a problem using sip behind PFSense but you did not specify if you are talking about a separate asterisk server or just IP endpoints that register to an external
SIP server..I do not have ANY problems when I test with just phones behind it (NAT) without any port forward rules that works just fine.
I'm really doing the opposite of that. Got an asterisk box behind it that needs to register to external places as well as allow static inbound on UDP port 5060 to allow others to register to me.
It all works perfect on every other firewall I use and test except PFSense. :-(
-
10 18.558549 4.4.4.4 5.5.5.5 SIP/SDP 1070 Status: 200 OK, with session description
What happens if you make Source = ANY?
-
I'm not sure exactly what you are asking because when defining a NAT port forward Source = ANY by DEFAULT.
-
I just completely and entirely re-did my setup without anyvirtual machines and using actual hardware servers with actual nics and am having the exact same problem.
I'm pretty convinced at this point that this is a serious bug and I'll need to report it.I'm asking the box to simply port forward UDP 5060 and it should port forward every single UDP DST port 5060 packet period and it is not.
It's appearing to be doing some kind of sip or otherwise inspection and not passing or natting the packet to the lan interface and it should absolutely NOT be working this way.
It's also not logging any dropped packets or reasons for not forwarding the packets as it occurs.
The inbound packet looks fine and is addressed to the WAN address with a destination UDP port 5060.
All other packets are making it across and appearing natted onto the LAN but these particular SIP ack packets are not.If I tell it to portforward UDP 5060 from anywhere to the LAN destination 10.73.73.8 UDP 5060 it should do this and it is not.
This is VERY broken. and is stopping me from being able to use the asterisk box with PFSense which is incredibly frustrating right now.Thank you again very much for trying to help.
:-)
-
I think what I'm running into here is that PFSense is trying to be WAY too application or WAY too SIP aware and making choices not to route/nat inbound based on various crap.
I'm now running into a similar problem with inbound RTP packets..They're not getting natted!! (and it seems to be due to PFSense randomizing outbound source ports) which I should be able to deal with with static outbound settings).
I really should be able to for at least troubleshooting purposes be able to tell the firewall to blindly accept all UDP packets in a port range and just NAT them and forward them on to the LAN!
But it's making stupid decisions NOT to do this based on the state of OTHER connections to the same port or port ranges!!I can recreate the UDP port 5060 problem with ease..
It's happening when the SIP trunk provider sends me an ACK packet on port 5060 after I have registered from a random source port..
Firewall is determining that this packet is somehow not associated or 'off color' because it is not destined to my source port..
This is bullshit because my port forwarding rule should override this!! and I should be getting the packet passed on to my internal host.
PFSense is incorrectly acssociating this packet to the fact that it came from the same host I'm talking to sending me packets to port 32701
And ignoring /dropping any new packets coming from the same distant host that are not associated with the first SIP connection.
This is INSANE! (Or I am) :-)There really really REALLY should be a way to bypass this behaviour and portforward inbound packets to the right place unconditionally.
I'm starting to see a pattern or mentality where the dynamically created reachback rules are somehow overriding the static port forward rules in some cases
which is completey the opposite of how it should work.. a port forwarding put in place should ALWAYS work regardless of what's going on dynamically. -
Alright I seem to have it all working now.. just a LOT of working through it and pissing & moaning as I went along.
If I setup NAT static outbound for the host to force only outbound SRC port 5060 my inbound 5060 packets now work coming back but ONLY because of the dynamic pipe created by forcing it to use port 5060!
This is a HIDEOUS ugly and painful port forwarding bug a half which I shall report! :-)Obviously if I had a sip registration with a SRC port of 34211 and the ITSP goes and sends me their ACKS to port 5060 but everything else to port 34211 (this is something they should not be doing)
Of course with sip that's debatable.. it may not actually be improper but it's weird.
But if portworwarding were actually working properly (which it's not) I'd get that improper or unexpected port 5060 SIP ACK packet to the asterisk box (as I do with any other firewall doing port forwarding) and all would be happy & work. -
I think what I'm running into here is that PFSense is trying to be WAY too application or WAY too SIP aware and making choices not to route/nat inbound based on various crap.
Not true, UDP is UDP, SIP isn't handled any differently from any other UDP traffic, there is 0 SIP intelligence in the underlying firewall.
-
[WAN]
No. Time Source Destination Protocol Length Info
10 18.558549 4.4.4.4 5.5.5.5 SIP/SDP 1070 Status: 200 OK, with session descriptionYour rule example above seems to indicate you are restricting incoming from a single IP…
Under source- 4.4.4.4, make any... But may be moot now.
pfSense by default obfuscates the outgoing port. From what I know SIP client devices expect response from the same port that they sent to... IE 5060, 5080, (vonage-) 10000...
If your running a sip server you should have static port on outgoing connections from that server.
If you can use 1:1 NAT.
-
Not true, UDP is UDP, SIP isn't handled any differently from any other UDP traffic, there is 0 SIP intelligence in the underlying firewall.
Well Good!! That's the way it should be however I'm seeing that my configured rule is NOT being adhered to
I have a port forward rule that specifies to allow from any as suggested ans is obvious (default) by the way.
And not all packets are being sent on ..
My observation is that it appears to be conditional as to what previous dynamic connections have been
made to the host that I am not getting the packets natted to the internal host from..This: "
[WAN]
No. Time Source Destination Protocol Length Info
10 18.558549 4.4.4.4 5.5.5.5 SIP/SDP 1070 Status: 200 OK, with session descriptionIS NOT A RULE! it is just the packet I am recieving at the WAN that should be getting natted and passed to the LAN but is not. It's a Text export from wireshark.
-
This is still looking like a very serious bug.
I'm working on a test scenario where this can be duplicated and recreated in the lab
easily with hooking up a few machines or doing it all within a virtual environment between a couple of virtual machines simulating the scenario.
And I'll pass my test setup over to the developers so they can easily recreate and replicate + see the problem I am talking about.The issue is quite clear.. with a port forward rule specifying that all UDP from anywhere with a DST port of 5060 should come in be allowed natted and passed to a specific internal host, PFSense should simply do just that and it is not.
Depending on other (dynamic) connections inbound on other ports from the same external host that is sending me packets on UDP 5060 (Which I'm getting on the WAN) I'm seeing PFSense fail to nat and send these specific inbound packets to the LAN interface and as a result they never reach the internal host.
-
If I have time this weekend, I'll try to reproduce this.
It happens will all 5060 udp packages or some packages?
Just to be 100% sure we have same rdr nat
On Firewall -> nat -> Port Forward
-
It only happens with some specific packages.
If I have a dynamic SIP registration going out –>10.73.73.8:(src)5060:(dst)5060-->PFSMyPublicIP:43112:5060-->RemotePublic IP:43112:5060
Packets coming back as RemotePublicIP:5060:43112--->PFSMyPublicIP:43112:5060--->10.73.73.8:5060:5060 work good as expected.However if RemotePublicIP suddenly sends us another packet like this: RemotePublicIP:5060:5060 (instead or aside from) RemotePublicIP:5060:43112
The packet destined to port 5060 does not get nated and send on to 10.73.73.8 like it should be due to the port forwarding rule.
It's as if the dynamically created reachback rule is somehow overwriting the port forward rule which should NEVER happen :-)My test scenario is this:
From PFSense Fresh install Defaults
1. assign single static IP to WAN Interface.
2. assign single static IP to lan interface. (10.73.73.1)
3. assign default gateway and DNS address for WAN
4. DHCP disabled on lan & wan
5. Start with auto outbound rule make sure Internet access & nat Lan to outside is working
6. Enable NOT blocking bogon networks (not needed to recreate problem) I just do it for testing.
7. Internal asterisk box placed at 10.73.73.8 and using PFS as it's gateway.
8. Asterisk nat settings are set properly for internal network range and external public IP address in sip.conf
9 In PFS Set Port forward from anywhere UDP DST port=5060 gets rdirect target set to 10.73.73.8 redirect port UDP 5060
10. In PFS Set Port forward from anywhere UDP DST Ports =11200-11300 redirect to target 10.73.73.8 UDP DST 11200-11300 (this is for RTP and NOT needed to demonstrate port 5060 failure to nat).
In my case port forward 5060 fail issue can still be tested without without RTP flowing to the internal box You just simply won't have working audio has nothing to do with the issue reported.11. Have asterisk register to an external SIP provider will by default create a dynamic inbound port back to asterisk (THIS IS OK).
12. You are now registered with provider outbound port is UDP 5060 inbound port will be random 32119 for example.. Inbound SIP from registered provider will be on UDP port 32119.
(my provder doesn't care if we randomize or dynamically change the outbound port it still works)
13. Same provider now sends you usual expected packets on port 32119 for SIP messaging as expected and they reach the internal box just fine.
14. TEST: Now same provder sends us another SIP packet but instead they send it to UDP PORT 5060 instead of 32119.
15. PFSense gets the packet but does not nat it and does not pass it on to 10.73.73.8 regardless of our port forward rule that it should.I can provide more details and packet captures and screenshots of my PFSense setup if this will save you any time as well as allow you remote access to my test config via Private message email or a phone call.
Also just confirming the focus here is now that I'm siting a bug in port forwarding.
I did get my system to 'work' by Forcing static outbound on UDP 5060 just to get PFSense get me the packets coming back on PORT 5060 via a separate mechanism other than a port forward rule.
aside from port forwarding rules.. This made my system work but also setup other ugly limitations that will cause me grief later on for sure when I need other (random ports) to work in combination with
5060 to/from the same remote host. -
You need to rewrite the source port on your outbound SIP as is done by default. I suspect the outbound NAT state is picking up the traffic rather than the rdr (port forward), because you aren't rewriting the source port.
-
You need to rewrite the source port on your outbound SIP as is done by default.
Sorry I'm not understanding exactly what you are saying here.
If I use it in a default fashion while outbound sip is being rewritten I am seeing the problem with inbound packets to port 5060 which is wrong.
A remote host can send inbound packets to multiple ports there is nothing wrong with this.I suspect the outbound NAT state is picking up the traffic rather than the rdr (port forward), because you aren't rewriting the source port.
Still not understanding at all (sorry). And we are dealing with an inbound static rule so outbound nat state should not matter.I still have an expectation as it works on all other firewalls and that is if you define a port forward rule it will work regardless of what is going on
dynamically.I still see no reason at all why this is not portwarding under all conditions. That is the sole purpose of port forwarding. Seriously.. there's no reason for it NOT to work unless
there is a bug in the packet inspection and/or decision making here.. It's a brain dead simple port forwarding task and it's failing. -
CMB, it sounds like you are telling me what I need to do to make it work (which I have done) and have made it work awhile back.. maybe you already read this.. I'm not sure. I have done exactly that and it does work. But portworwarding itself in this case still does not behave the way I expect it should.
It breaks other things that I won't dig into right now unless you ask, as it will take this off topic.
I'll continue to listen & learn for awhile before I open a bug report in case I'm missing something. Thank you
so very much for helping out! -
I am having a similar problem but I can at least get it to work using Manual NAT. The problem with this is once I turn on manual NAT with Static, it breaks my other subnets web browsing capabilities.
Here is my scenario:
192.168.0.0 - Local lan
192.168.1.0 - VPN lan
192.168.12.0 - MPLS lan
192.168.13.0 - MPLS lan
192.168.14.0 - VPN lan
192.168.23.0 - VPN Lan
192.168.24.0 - VPN lan
192.168.25.0 - VPN lan
192.168.26.0 - VPN lanThe phone system sits on 192.168.0.6 and there are IP phones on 192.168.23.0 and 192.168.25.0 working fine. We had to add a SIP provider for routing "out of area" numbers to the phone system from the phones sitting on the 192.168.14.0 network. The intercom traffic works fine between 192.168.0.0 and 192.168.14.0 through the VPN. All HTTP traffic works fine through squid3 as a proxy from all networks. With this currently set as Auto Outbound NAT everything works fine except the SIP traffic. I can call the number, the packets get routed through the phone system and the phones at 192.168.14.0 ring. I can hear them answer but they cannot hear me. To resolve this I turned on Manual Outbound NAT. Fixed the phone issue, but now no one can access web pages through squid3 on any network other than 192.168.0.0 which is local to the firewall and the phone system.
I have come to the conclusion that the Static Port option breaks squid3 but fixes the phones, and in turn static mapping turned off fixes squid3 but breaks the phones (one way audio). We are using an outside provider so there are no SIP equipment onsite.
Any one have any suggestions or need further info please feel free. I'm at the point I'll try anything now.
