RTP UDP VOIP ports forward: packet rewritten, pass logged, lost! [solved-ish]

hcoin

Short version: A nat rule that specifies many UDP ports to forward logs packets as 'passed' – but never sends them out after having received them.

I've spent a day trying to solve this, no luck. I have a nat port forward rule
2.1 amd64:
If Proto Src. addr Src. ports Dest. addr Dest. ports NAT IP NAT Ports Description
add
change INET1MC UDP * * 97.64.213.58 15000 - 15400 192.168.50.78 15000 - 15400

On the wan interface this packet arrives.

17:31:04.862393 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 200)
70.194.133.137.5944 > 97.64.213.58.15348: [udp sum ok] UDP, length 172

I turn on the 'log' option on the associated rule in the filter, and there the packet is logged as having arrived.

pass
Oct 16 17:29:15 INET1MC 70.194.133.137:5944 Easy Rule: Pass this traffic 192.168.50.78:15348 UDP

No packet having the named source address ever is logged as leaving the LAN interface. No entry in the syslog for having that packet blocked. Tried 'pure nat' tried 'proxy' no joy. Natting does work as there are sip phones that register correctly using single port tcp/udp nat.

What am I missing?

What could possibly correctly rewrite the destination correctly, log the packet and then – drop it?

What is different about a range of 400 ports? It's as if the router just eats these packets without comment. It's not an artifact of pf's packet capture on the lan, wireshark on the destination interface shows outbound rtp traffic only, no rtp incoming packets from the router at all, on any port. (plenty of sip tcp/udp packets on other ports from the router. sip 'registrations' succeed.

hcoin

I look at the rules and even though I specified the nat was for one interface interface in the GUI, the rules for rediecting the UDP ports are there for all the interfaces, even ones having nothing to do with the rules at all.

rdr on em1 inet proto udp from any to 97.64.213.58 port 15000:15400 -> 192.168.50.78
rdr on em0 inet proto udp from any to 97.64.213.58 port 15000:15400 -> 192.168.50.78
rdr on re2 inet proto udp from any to 97.64.213.58 port 15000:15400 -> 192.168.50.78
rdr on re0 inet proto udp from any to 97.64.213.58 port 15000:15400 -> 192.168.50.78

I'll list it as a bug.

rbgarga

I tried to reproduce it on a 2.1-RELEASE without success.

Could you share your config.xml (removing sensitive data) with me?

hcoin

Well, the puzzle certainly deepens. Seems some UDP packets to the same destination and allowed port range get through, while others do not.
Both the following packets are logged as 'Pass'. The first does not get sent out the LAN port. The second does. What could possibly be the difference? Both are logged as 'pass' by pfsense.

This packet never made it out the lan port, though logged as 'pass' as incoming from the wan. The PF packet dump correctly shows the destination rewritten to the private rdr lan address. After this, a netcat generated packet to the same port range actually does make it through to the lan side. What possibly could cause one to not be let out the lan, while the other has no problem?

Fails to make it out the lan:

No. Time Source Destination Protocol Length Info
177 4.285587 70.194.133.137 97.64.213.58 RTP 214 PT=ITU-T G.711 PCMU, SSRC=0x41E33244, Seq=10486, Time=3106902162

Frame 177: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Oct 16, 2013 14:54:01.275346000 Central Daylight Time

rbgarga

As Chris mentioned when close the redmine ticket you opened, this is not a bug, you have NAT reflection set, and this is how it works.

rbgarga

@Renato:

As Chris mentioned when close the redmine ticket you opened, this is not a bug, you have NAT reflection set, and this is how it works.

If you don't want the NAT rules on other interfaces, set Nat Reflection to "Disabled" instead of "Pure NAT"

hcoin

Thanks, I suggested a language change as 'pure nat' led me to think it was going to 'purely nat' the interface I asked for and not all the rest. Even after doing the above, notice that the heart of the problem remains:

Two UDP packets logged as 'pass' by PF cause one to be let out the lan port, while the other is dropped.

Both are to the same destination IP, both fall within the allowed port range, both are logged as 'pass'. One makes it out the lan, the other gets dropped with no log entry. I don't get it. Any clue at all?

hcoin

The only difference I see is one packet has "Padding: 6f0a6f0a6f0a6f0a6f0a6f0a6f0a" (the one passed but never let out the lan port) and the other does not.

hcoin

Update: setting net.inet.udp.log_in_vain=1 does not log any of the wan udp packets listed as 'pass' but that nevertheless are not emitted from the lan.

What are the conditions under which pfsense will log a inet4 udp packet as 'pass' in the firewall log, but not actually send it out the lan port?

Seriously I have no good direction to continue hunting here short of setting up a freebsd kernel debugging session and then trying to follow the packet flowing through the kernel. Surely there must be a better way. Any clue at all?

The two packets are set above. Same allowed port range, same destination ip, both logged as 'pass', one sent out the lan, the other not. I hope this is some major stupid oops on my part. I'm willing to look all kinds of embarrassed chump if only there was a simple answer.

hcoin

Disabling hardware checksum offload did not help.

hcoin

More clues, leaving me totally in the dark (from pfctl -s all). Still logging as 'passed' the padded udp packet above, but not letting it out the lan port, while the netcat one sails through to the same dest via the wan.

rdr on em1 inet proto udp from any to 97.64.213.58 port 15000:15450 -> 192.168.50.78

pass in quick on em0 inet proto udp from any to 192.168.50.78 port 14999 >< 15451 keep state label "USER_RULE: Allow RTP traffic"

em1 udp 192.168.50.78:15091 <- 97.64.213.58:15091 <- 70.194.136.230:5445 NO_TRAFFIC:SINGLE
em0 udp 70.194.136.230:5445 -> 192.168.50.78:15091 SINGLE:NO_TRAFFIC
em1 udp 192.168.50.78:5060 <- 97.64.213.58:5060 <- 70.194.136.230:14476 MULTIPLE:MULTIPLE
em0 udp 70.194.136.230:14476 -> 192.168.50.78:5060 MULTIPLE:MULTIPLE
em0 udp 70.194.136.230:5446 <- 192.168.50.78:15350 NO_TRAFFIC:SINGLE
em1 udp 192.168.50.78:15350 -> 97.64.213.58:40345 -> 70.194.136.230:5446 SINGLE:NO_TRAFFIC
em1 udp 192.168.50.78:15351 <- 97.64.213.58:15351 <- 70.194.136.230:5447 NO_TRAFFIC:SINGLE
em0 udp 70.194.136.230:5447 -> 192.168.50.78:15351 SINGLE:NO_TRAFFIC

USER_RULE: NAT Forward RTP 1171 10 1020 5 440 5 580

(em0->lan, em1->wan).

hcoin

Skipping right over all the steps that cost me about three days and getting to the solved-ish process:

VOIP / RTP UDP media forwarding on master/backup pfsense setup:

IF your setup is a primary/backup two pfsense box failover operation:
AND you change a NAT port forward rule OR related filter rule
AND RTP VOIP traffic isn't working (no audio) , WHERE the SIP / PBX / VOIP phones say the call is 'in progress'
AND you see RTP traffic showing up as incoming properly addressed on the WAN side packet capture.
AND you see RTP traffic showing up as outgoing properly addressed on the LAN side packet capture.
BUT you don't see the traffic crossing pfsense and getting out the other interface, despite firewall logs saying 'pass' on the packets involved, but with packet captures showing those packets pfsense never actually sends:

THEN:

Delete the nat port forward rule for the UDP packets. Delete any associated filter rule.
Go to System/Advanced Firewall/NAT/ NAT Reflection mode for port forwards. Set that to disable. Save it.
Go to the backup pfsense box. Do the same thing. No, pfsync doesn't seem to sync this item.
Go to any other non-RTP 'port forward' rules you may have and set the reflection mode there to what you would prefer.
in English: 'Use system default' means traffic coming in over the wire to the interface named in the gui that matches the rule will have its destination re-written as you desire. No other interface will be affected. Due to system quirks traffic that is coming in the interface containing the forward (re-written) address that matches the rule will be lost. You want this for RTP if you are connected to any PBX system that is programmed to deal with NAT and SIP – but wait, don't enter it at this step.
'Enable Pure NAT' -- Does the above, but also puts in a rule so that traffic coming in to ANY interface heading for the named destination in the rule gets re-written to go to the given address, with replies (hopefully) going out the interface named in the GUI for the rule. Once again, owing to system quirks, traffic coming in the interface that holds the final rewritten destination will be lost if it's destination address matches wan/incoming rule. Use only if you are really sure interfaces now or in the future, virtual, vpn or actual, that ever show up on this box will want those ranges forwarded to the named destination, and possibly routed magically in reply in wondrous and unexpected ways that don't lead to being able to talk on the phone.
Enable Nat+Proxy -- Does the above but under verrry special conditions that require all the forwarding when taken together using this feature to not exceed port quantity limits, will do the right thing when traffic arrives at the destination interface that matches the rule, sending it back out that interface. This is the default. God help you if you put in too many ports in one rule or in combination. Add one, increase the range of one, and get out your calculator and tally them all to stay under the limit. If you can think through how you use pfsense so as to avoid sending traffic 'in' to interfaces that are just going to send it right back out the same wire you will be better off and your pfsense implementation will scale to great traffic sizes with more ease. Using dns forwarding so internal lookups give the local destination is one idea.

Back to VOIP

Go to Diagnostics / States / Reset States on the primary. Hit the reset button. Wait.
Reboot the primary and the backup. Wait.

7) Add the 'port forward' rule for your RTP range. Check the 'enable static port' button. Choose 'use system default' for 'nat reflection. Enter one port forward rule for every ISP interface you have. Be sure you set your SIP PBX to limit itself to the RTP range of ports you've chosen. Basically think one port per ongoing conversation. Configure your PBX's 'internal' configuration to send traffic that never needs to leave private address spaces to send calls directly to the pbx/soft phone never using public ip's. PFsense will route to the various vpns and internal subnets correctly.

What a ride!