Outbound NAT not being applied even though rules are there
-
Hi,
I have a setup with two dedicated servers running pfSense 2.4.4p3 in a CARP environment.
The Firewall are connected for xmlrpsync and pfsync with a dedicated copper link without active components in between.
The interfaces itself run over two Intel X710 in each machine. Two of the four ports are bonded into a LAG which again uses VLAN 802.1Q to seperate the networks from each other.MOST is working great but we're struggling with some connections not being established. While searching I found out that some packets leave the firewall on the WAN interface (LAGG0.209) with a private source IP set. This link helped me figure it out.
The thing is not every client on the network isn't beign NATed only some and only sometimes. I can't find the reason, why and when.
Those are the networks that suffer this problem:
<opt6> <descr><![CDATA[RXMGMTCPE]]></descr> <if>lagg0.217</if> <enable></enable> <ipaddr>192.168.32.2</ipaddr> <subnet>22</subnet> <ipaddrv6>2a00:aaaa:0:10d::2</ipaddrv6> <subnetv6>64</subnetv6> <blockbogons></blockbogons> <spoofmac></spoofmac> </opt6> <opt14> <descr><![CDATA[RXVOIP_SL1]]></descr> <if>lagg0.225</if> <enable></enable> <blockbogons></blockbogons> <spoofmac></spoofmac> <ipaddr>192.168.38.2</ipaddr> <subnet>24</subnet> <ipaddrv6>2a00:aaaa:0:10b::2</ipaddrv6> <subnetv6>64</subnetv6> </opt14>I set Outbound NAT to manual since I want static ports. Those is the corresponding configurations:
<rule> <source> <network>192.168.32.0/22</network> </source> <sourceport></sourceport> <descr><![CDATA[user created rule for ISAKMP - RXMGMTCPE to WAN]]></descr> <target>93.111.250.180</target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <ipprotocol>inet</ipprotocol> <destination> <any></any> </destination> <dstport>500</dstport> <updated> <time>1576921670</time> <username>user@2a00:aaaa:0:102::1001 (RADIUS/auth.domain1.net)</username> </updated> <created> <time>1576921670</time> <username>user@2a00:aaaa:0:102::1001 (RADIUS/auth.domain1.net)</username> </created> </rule> <rule> <source> <network>192.168.32.0/22</network> </source> <sourceport></sourceport> <descr><![CDATA[user created rule - RXMGMTCPE to WAN]]></descr> <target>93.111.250.180</target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <ipprotocol>inet</ipprotocol> <destination> <any></any> </destination> <updated> <time>1576921738</time> <username>user@2a00:aaaa:0:102::1001 (RADIUS/auth.domain1.net)</username> </updated> <created> <time>1576921738</time> <username>user@2a00:aaaa:0:102::1001 (RADIUS/auth.domain1.net)</username> </created> </rule> <rule> <source> <network>192.168.38.0/24</network> </source> <sourceport></sourceport> <descr><![CDATA[Auto created rule for ISAKMP - RXVOIP to WAN]]></descr> <target>93.111.250.180</target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <dstport>500</dstport> <created> <time>1569411785</time> <username>Manual Outbound NAT Switch</username> </created> <updated> <time>1574335157</time> <username>admin@2a00:aaaa:0:108:3dd:6038:2023:d7f2 (Local Database)</username> </updated> </rule> <rule> <source> <network>192.168.38.0/24</network> </source> <sourceport></sourceport> <descr><![CDATA[Auto created rule - RXVOIP to WAN]]></descr> <target>93.111.250.180</target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <created> <time>1569411785</time> <username>Manual Outbound NAT Switch</username> </created> <updated> <time>1576595525</time> <username>user2@2a00:aaaa:0:108:3dd:6038:2023:d7f2 (RADIUS/auth.domain1.net)</username> </updated> </rule>This is an example of my tcpdump
tcpdump -ni lagg0.209 net 10 or net 192.168 or net 172.16.0.0/12 18:03:04.547170 IP 192.168.32.127.1026 > 5.111.140.18.1812: RADIUS, Access-Request (1), id: 0x56 length: 80 18:03:07.982801 IP 192.168.38.152.2051 > 93.111.248.15.123: NTPv4, Client, length 48 18:03:10.505512 IP 192.168.38.151.2051 > 93.111.248.15.123: NTPv4, Client, length 48 18:03:17.982789 IP 192.168.38.152.2051 > 93.111.248.15.123: NTPv4, Client, length 48 -
Those rules don't show that the traffic should be natted... So why should it be natted?
<target>93.111.250.180</target> -
@johnpoz said in Outbound NAT not being applied even though rules are there:
Those rules don't show that the traffic should be natted... So why should it be natted?
<target>93.111.250.180</target>I don't know why the OP chose to post the xml instead of the GUI rules, but that appears correct . Source net is the internal subnet, target is the external IP.
-
But is that the carp vip? He is in a carp setup.. What are his rules..
Much easier to figure out what is going on from actual screenshots and full picture of setup vs tiny snip of info..
-
Yes, it's one of the CARP ip which is being used by multiple local nets. I've attached two screenshots of my configuration.
WAN CARP status on Master device:
Outbound NAT rules:
This is neither the full CARP list nor the full Outbound NAT list, since it's too long to get into one screenshot.
Edit:
Here's also what pfctl shows:pfctl -sn ... nat on lagg0.209 inet from 192.168.32.0/22 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.32.0/22 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.38.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.38.0/24 to any -> 93.111.250.180 static-port ... -
What I today tried is to switch over to secondary firewall. We suffer the same situation there.
Then I rebooted the first firewall and switched back to it. Sadly, no improvement afterall.
Here's the full output of pfctl:[2.4.4-RELEASE][root@fw2-rx.domain1.net]/root: pfctl -sn no nat proto carp all nat-anchor "natearly/*" all nat-anchor "natrules/*" all nat on enc0 inet from 10.254.252.0/24 to 172.24.0.0/16 -> 192.168.222.1 nat on enc0 inet from 192.168.20.0/24 to 172.24.0.0/16 -> 192.168.222.1 nat on lagg0.209 inet from 192.168.13.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.13.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 127.0.0.0/8 to any port = isakmp -> 93.111.250.178 static-port nat on lagg0.209 inet from 127.0.0.0/8 to any -> 93.111.250.178 port 1024:65535 nat on lagg0.209 inet6 from ::1 to any port = isakmp -> 2a00:aaaa:0:16::2 static-port nat on lagg0.209 inet6 from ::1 to any -> 2a00:aaaa:0:16::2 port 1024:65535 nat on lagg0.209 inet from 192.168.25.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.25.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.89.0/24 to any port = isakmp -> 93.111.250.181 static-port nat on lagg0.209 inet from 192.168.89.0/24 to any -> 93.111.250.181 static-port nat on lagg0.209 inet from 172.16.0.0/22 to any port = isakmp -> 93.111.250.178 static-port nat on lagg0.209 inet from 172.16.0.0/22 to any -> 93.111.250.178 static-port nat on lagg0.209 inet from 192.168.16.0/22 to any port = isakmp -> 93.111.250.178 static-port nat on lagg0.209 inet from 192.168.16.0/22 to any -> 93.111.250.178 static-port nat on lagg0.209 inet from 192.168.12.0/24 to any port = isakmp -> 93.111.250.22 static-port nat on lagg0.209 inet from 192.168.12.0/24 to any -> 93.111.250.22 static-port nat on lagg0.209 inet from 192.168.20.0/24 to any port = isakmp -> 93.111.250.22 static-port nat on lagg0.209 inet from 192.168.20.0/24 to any -> 93.111.250.22 static-port nat on lagg0.209 inet from 192.168.102.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.102.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.38.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.38.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.26.0/24 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.26.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.98.0/23 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.98.0/23 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 10.254.252.0/24 to any port = isakmp -> 93.111.250.22 static-port nat on lagg0.209 inet from 10.254.252.0/24 to any -> 93.111.250.22 static-port nat on lagg0.209 inet from 192.168.52.0/23 to any port = isakmp -> 93.111.250.182 static-port nat on lagg0.209 inet from 192.168.52.0/23 to any -> 93.111.250.182 static-port nat on lagg0.209 inet from 192.168.40.0/23 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.40.0/23 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.32.0/22 to any port = isakmp -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.32.0/22 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.10.0/24 to any -> 93.111.250.180 static-port nat on lagg0.209 inet from 192.168.250.0/24 to any -> 93.111.250.180 static-port no rdr proto carp all rdr-anchor "relayd/*" all rdr-anchor "tftp-proxy/*" all rdr on lagg0.209 inet proto tcp from <H_RX_asterisk> to 93.111.250.183 port = https -> <H_vws> round-robin rdr on lagg0.209 inet proto tcp from <H_RX_ip_domain1_net> to 93.111.250.183 port = https -> <H_vws> round-robin rdr on lagg0.209 inet proto tcp from <H_RX_www_domain1_net> to 93.111.250.183 port = https -> <H_vws> round-robin rdr on lagg0.209 inet proto tcp from <H_RX_www_domain2_de> to 93.111.250.183 port = https -> <H_vws> round-robin rdr-anchor "miniupnpd" allI can't find any errors. Might this be a bug?
-
Does somebody have an idea? Otherwise I would open a bug report because there doesn't seem to be configuration errors.
-
So you trying to do static port? Why?? You understand that can't work without issues as soon as you have more than one client behind the nat right... As some point you will have a conflict..
You have how many clients sharing that same public IP? And you want the ports to be static - how does that work, when there is a conflict?
-
@johnpoz
Thank you. I'm aware of this page:
https://docs.netgate.com/pfsense/en/latest/nat/static-port.htmlFor testing I disabled static port for a local subnet that suffers from the problem but I can still see local source IPs leaving WAN.
What's weird is that most of the time it's only NTP that is suffering this problem and most of the times it's only two hosts that are not being NATed.14:14:54.088287 IP 192.168.38.151.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:14:58.725877 IP 192.168.38.152.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:15:04.088357 IP 192.168.38.151.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:15:08.725876 IP 192.168.38.152.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:15:14.088417 IP 192.168.38.151.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:15:18.725892 IP 192.168.38.152.2051 > 93.111.248.15.123: NTPv4, Client, length 48 14:15:24.088503 IP 192.168.38.151.2051 > 93.111.248.15.123: NTPv4, Client, length 48 -
And does that source port have any other nats assigned to it, because you were using static? Why did you only turn if off on some of your source networks? That doesn't solve the problem of a conflict when you have 1 IP, with only X number of ports to use, and multiple clients behind the nat picking source ports... At some point they are going to do the same source port... Then what happens? Maybe it goes out un natted..
What you linked too doesn't go into the problem with static ports and multiple clients..
What happens when client X wants to use source port Y, and then client Z also wants to use source port Y for its connection.. How does that work?? You can not maintain that nat..
-
I disabled "static port" for every Manual Outbound NAT except for those ISAKMP entries but still see local IPs on WAN.
-
did you clear the old states?
-
That seems to have worked, thank you so much.
I thought that the combination of Source IP, Destination IP, Source Port and Destination Port were to determine one specific connection but when the answers to my outbound traffic arrive at the firewall there are no internal IP in the headers (of course). -
I have never found the time to do full testing of what "could" or what actually should happen when you force static mappings like that and you run into a conflict..
While your correct a state would have the dest IP, so you could get by with the returning traffic source being different..
But with enough clients you could run into same dest and same source port.. Now your hosed!!
Using static ports is fine when you have a specific 1 application in mind sort of thing... But when you try to do that with a large population of clients for "all" the ports - yeah your at some point going to run into issue..
Glad you got it sorted..
-
We use pfSense for another installation with Sony Playstation clients. Those devices are really stubborn when it comes to port randomization, they just wont work with it. That's the reason why we made it the default which doesn't seem to be a good standard.
Cisco seems to be doing a mix of both with iOS. They use static ports as long as there's no conflict and only if they detect one, they gamble a different source port.
