Incoming packets from single source bypassing 1:1 NAT?

[ScottCall]

I have a 2xXG-1541 cluster running 2.4.5p1 (going to go to 21.05.01 soon) and have a weird issue.

I have an IP PBX on a DMZ vlan behind the pfsense. I have a 1:1 NAT for one of the addresses I have from my provider. It's a new installation so for testing I have a permit all rule with the internal IP as the destination and logging enabled.

The 1:1 NAT works for the most part but one of my two SIP trunk providers seems to be able to bypass that NAT?

For all traffic to this IP, it hits the permit all rule and gets logged just fine, with the exception of the IP(s) from the secondary SIP provider.

Packets (mostly udp/5060 because SIP) get logged by the default deny and show the public IP address as the destination.

I've checked and cleared any states mentioning the SIP provider IPs and it doesn't change the behavior.

Here is an example of the firewall log. The top two lines were sent using netcat, the bottom two from the SIP provider. Bother were to the same destination (ending in .173) and the ones from netcat show the correct internal destination but the ones from the SIP provider only show the public IPs?

I've also compared another 1:1 NAT with pfctl and the nat setups looks identical (except for the IPs of course)

Any suggestions short of rebooting?

Thanks!

[johnpoz]

@scottcall said in Incoming packets from single source bypassing 1:1 NAT?:

but the ones from the SIP provider only show the public IPs?

Well it was blocked.. So yeah..

Its really hard to help with out seeing the rules.. But not allowed, then your rules are not triggering, or a rule above is blocking, etc. Really need to see the rules..

[ScottCall]

@johnpoz said in Incoming packets from single source bypassing 1:1 NAT?:

@scottcall said in Incoming packets from single source bypassing 1:1 NAT?:

but the ones from the SIP provider only show the public IPs?

Well it was blocked.. So yeah..

Its really hard to help with out seeing the rules.. But not allowed, then your rules are not triggering, or a rule above is blocking, etc. Really need to see the rules..

The blocked packets are blocked by default deny but are to the same destination public IP as the packets that correctly hit the permit rule for all. That's the conundrum. If I explictly allow that source/destination pair that is being blocked, it no longer shows blocked but still never is NAT'd.

My understanding of the order of operations was NAT first then Rules, so the 1:1 NAT of the public IP to internal IP and then the permit-all to the internal IP would apply?

Here is the 1:1 NAT Rule:

And here is the incoming firewall rule:

All packets to the external address in the NAT rule will be translated and passed to the firewall, which should then pass them to the internal host because it's permit all, right? In my testing, several hosts (Primary SIP provider, my off-site colo, my home network) can all hit the public IP address and show up on the destination host (and in the pfsense logs) only connections from this second SIP provider are not getting translated, and therefore getting blocked.

Here is how the rules show up in pfctl -s all (X.Y.Z.173 is external, A.B.1.73 is internal, A.B.1.251 is the pfsense IP and A.B.1.1 is the CARP Virtual IP default gateway shared) ix0 is the Internet and ix1.600 is the DMZ Vlan that the destination host is on.

pfctl -sr | grep A.B.1.73:

pass in log quick on ix0 reply-to (ix0 WANIP) inet from any to A.B.1.73 flags S/SA keep state label "USER_RULE: 3CX Incoming"

pfctl -sn | grep X.Y.Z.173:

rdr on igb0 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.600 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.400 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on igb1 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.800 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on enc0 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on openvpn inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
binat on ix0 inet from A.B.1.73 to any -> X.Y.Z.173

pfctl -sn | grep A.B.1.73:
no nat on ix1.600 inet from (ix1.600) to A.B.1.73
nat on ix1.600 inet from A.B.1.0/24 to A.B.1.73 -> A.B.1.251 port 1024:65535
no nat on ix1.600 inet from (ix1.600) to A.B.1.73
nat on ix1.600 inet from A.B.1.0/24 to A.B.1.73 -> A.B.1.251 port 1024:65535
rdr on igb0 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.600 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.400 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on igb1 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on ix1.800 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on enc0 inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
rdr on openvpn inet from any to X.Y.Z.173 -> A.B.1.73 bitmask
binat on ix0 inet from A.B.1.73 to any -> X.Y.Z.173

[johnpoz]

@scottcall dude post up picture of your rules.. Nobody wants to look at that. And you hiding your internal rfc1918 address??

Post UP the full listing of your rules.. Not just 1 thing with everything freaking hidden..

Here is the thing... If its denied by default rule - then your allow didn't trigger, period.

If a nat hits, then it looks to see if firewall rule allows it. Your being allowed is clearly from a different source IP.. Which triggers..

[ScottCall]

@johnpoz said in Incoming packets from single source bypassing 1:1 NAT?:

@scottcall dude post up picture of your rules.. Nobody wants to look at that. And you hiding your internal rfc1918 address??

Post UP the full listing of your rules.. Not just 1 thing with everything freaking hidden..

Here is the thing... If its denied by default rule - then your allow didn't trigger, period.

If a nat hits, then it looks to see if firewall rule allows it. Your being allowed is clearly from a different source IP.. Which triggers..

I assure you I'm not trying to be obtuse I'm trying to not accidentally share something compromising/exploitable about my network (and yes I do know that security through obscurity sucks, but I'd rather not take unnecessary risks)

Here are the 1:1 NAT rules with the public IP redacted.

Here are all of the firewall rules on that same "ZAYO" Interface with minimal redaction:

If there's a better way to get this to you than screenshots, let me know.

The external IP ending in .173 is not listed on any other NAT section (outgoing or port forwarding) and the only other rule that covers the internal address (172.17.1.73) is an allow all TCP/UDP on the DMZ interface to the Internet. If needed I can redact and post those or DM you with them if it's helpful. The same goes for any data from pfctl, the only listing for the external .173 and internal 1.73 is included in what I've said above (I was worried an old rule might've been stuck somewhere?)

Since it's well after hours now, I reset the state tables on the firewall just to see if something was stuck and it did not remedy the situation. My next thought is to try a full restart but due to some past experiences, I'd rather be in the office when that happens than remote, so it will have to wait a couple of days.

This is why I think I might have found a glitch or a bug instead of just fat fingering something, although I am 100% open to it being something I messed up.

I do appreciate your time and hope this helps to clarify, feel free to ask for anything else that might be helpful.

Thanks

[johnpoz]

@scottcall that is much easier to read.. ;)

But you have a lot of block rules, where are you showing that its the default deny rule.. Turn rule names in your firewall log so you can see which rule is actually blocking. You have a lot of block rules above that don't even log, etc.

From your rules, clearly you have some that are not 1:1 nat - so what are the other port forwarding rules?

[ScottCall]

@johnpoz Thanks

Here is the log snippet that shows the packets from the SIP provider being blocked by "Default deny rule IPv4" with the external IP showed as the destination (since it did not get 1:1 NAT) but then 10 seconds later I send random UDP from my home network to the same external IP (also using same source and dest port) and it makes it through.

As you said there are other (port forwarding) incoming NAT setup, but none of them have either the source, external destination or internal destination listed. Here they are for good measure:

At this point unless you're spotting anything I obviously mucked up, it'll hold until I can schedule a reboot in the next couple days and if after a reboot (and maybe update to the latest, which after 2.4.5 caused a ton of problems for me at launch is a bit scary) and see if that improves anything.

Thanks

[ScottCall]

Okay so things are stranger and my mind is hitting a wall.

I did the upgrade to 21.05.1 and it went super smooth (thanks Netgate!)

But!

I was still having the issue were traffic from a single IP address was not getting processed in 1:1 NAT.

Same as I saw in 2.4.5p1, literally any IP on the internet except the one from my SIP provider would be properly NAT'd and send through to the 3CX system.

Grasping at straws I was wondering if the state created by the 3CX registering with the provider was an issue since it contained the same IP and port info as the incoming connection? (Blue is my public address, Red is the SIP provider's)

Just for grins, I changed the trunk time at both ends to be IP based (no authentication) just to see if anything changed.

For reasons I cannot comprehend, it started working.

SAME source address, SAME destination address, but it's being properly NAT'd now.

I literally have no idea why that worked when the other way (registration based) didn't?

So I guess everything is okay now but I really really hate problems that don't make any sense and the resolution just feels like pushing off the inevitable when it breaks again.

Thank you for your help, and if there's anything I've posted above that catches your eye, please let me know, otherwise I will have to be half-satisfied that it works but half-unsatisfied because there's no logical reason for it to have not worked in the first place.