Outbound NAT not working for single host (multi-WAN)

clarknova

pfsense 2.5.1

This firewall is dual-WAN with virtual IPs on both WAN interfaces.

 PUB03 (wan)     -> vmx1       -> v4: x.y.z.43/26
 LAN (lan)       -> vmx0       -> v4: 10.7.24.254/24
 PUB01 (opt1)    -> vmx2       -> v4: a.b.c.86/26

ifconfig vmx2|grep a
	inet a.b.c.86 netmask 0xffffffc0 broadcast a.b.c.127
	inet a.b.c.116 netmask 0xffffffc0 broadcast a.b.c.127
	inet a.b.c.69 netmask 0xffffffc0 broadcast a.b.c.127
	inet a.b.c.78 netmask 0xffffffc0 broadcast a.b.c.127
	inet a.b.c.115 netmask 0xffffffc0 broadcast a.b.c.127

Multiple Port Forward rules are configured on the various public addresses and multiple outbound NAT rules are configured. For the most part, everything works as expected, but there is one exception that I haven't been able to figure out. I've attached screenshots of the port forward and outbound NAT rules that pertain to this host. For reasons I haven't been able to nail down, when I telnet to a.b.c.69 on port 46300, the reply exits the PUB01 interface with the local IP address intact like this:

tcpdump -nivmx2 port 46300
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx2, link-type EN10MB (Ethernet), capture size 262144 bytes
16:51:47.883040 IP q.r.s.t.48204 > a.b.c.69.46300: Flags [S], seq 2523553336, win 64240, options [mss 1460,sackOK,TS val 680265598 ecr 0,nop,wscale 7], length 0
16:51:47.886255 IP 10.7.24.60.46300 > q.r.s.t.48204: Flags [S.], seq 3792540171, ack 2523553337, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 699027 ecr 680265598], length 0

What am I doing wrong here?

edit: One further note, if I 'telnet a.b.c.69 46300' from a host on the same subnet as the PUB01 interface, the reply gets its source address rewritten as expected and the connection succeeds.

clarknova

I disabled the port forward and outbound NAT rules for 10.7.24.60 and enabled 1:1 NAT without changing the firewall rules, but I'm still seeing the same thing, which is non-NAT packets from this host on the PUB01 interface. Here is a dump of my NAT rules:

pfctl -sn 
no nat proto carp all
nat-anchor "natearly/*" all
nat-anchor "natrules/*" all
nat on vmx2 inet from 10.7.24.101 to any -> a.b.c.116 port 1024:65535
nat on vmx2 inet from 10.7.24.102 to any -> a.b.c.115 port 1024:65535
nat on vmx2 inet from 10.7.24.125 to any -> a.b.c.86 port 1024:65535
nat on vmx1 inet from 10.7.24.126 to any -> x.y.z.28 port 1024:65535
nat on vmx2 inet from 10.7.24.127 to any -> a.b.c.78 port 1024:65535
nat on vmx1 inet from 10.7.24.135 to any -> x.y.z.31 port 1024:65535
nat on vmx1 inet from 10.7.24.0/24 to any -> x.y.z.43 port 1024:65535
nat on vmx1 inet from (self) to any port = isakmp -> x.y.z.43 static-port
nat on vmx1 inet6 from (self) to any port = isakmp -> (vmx1) round-robin static-port
nat on vmx2 inet from (self) to any port = isakmp -> a.b.c.86 static-port
nat on vmx2 inet6 from (self) to any port = isakmp -> (vmx2) round-robin static-port
no rdr proto carp all
rdr-anchor "tftp-proxy/*" all
rdr on vmx2 inet proto tcp from any to a.b.c.116 port = 46300 -> 10.7.24.101
rdr on vmx2 inet proto tcp from any to a.b.c.116 port = 51502 -> 10.7.24.101
rdr on vmx2 inet proto tcp from any to a.b.c.116 port 7137:7140 -> 10.7.24.101
rdr on vmx1 inet proto tcp from any to x.y.z.43 port 7137:7140 -> 10.7.24.100
rdr on vmx1 inet proto tcp from any to x.y.z.43 port = 46300 -> 10.7.24.100
rdr on vmx1 inet proto tcp from any to x.y.z.43 port = 51502 -> 10.7.24.100
rdr pass on vmx2 inet proto tcp from any to a.b.c.86 port = http -> 10.7.24.125
rdr pass on vmx2 inet proto tcp from any to a.b.c.86 port = https -> 10.7.24.125
rdr-anchor "miniupnpd" all
binat on vmx2 inet from 10.7.24.101 to any -> a.b.c.116
binat on vmx1 inet from 10.7.24.126 to any -> x.y.z.28
binat on vmx2 inet from 10.7.24.60 to any -> a.b.c.69
binat on vmx2 inet from 10.7.24.127 to any -> a.b.c.78
binat on vmx2 inet from 10.7.24.102 to any -> a.b.c.115
binat on vmx1 inet from 10.7.24.135 to any -> x.y.z.31

I don't see any problem here, but maybe somebody else does.

viragomann

@clarknova
Not really clear what you're trying to achieve here.
10.7.24.60 is in you LAN. When you forward packets from PUB01 to it, the packets won't go out on PUB01 but on LAN. So the outbound NAT rule on PUB01 for the sourece 10.7.24.60/32 is quite useless.

clarknova

@viragomann, PUB01 is on the internet. Forwarding packets from PUB01 to 10.7.24.60 is working fine, ie, the destination address is rewritten to 10.7.24.60 as expected, but when 10.7.24.60 replies, the reply is forwarded back out of PUB01, which is also expected. What is not expected is that the source address of the reply is not rewritten to the public virtual IP of PUB01, a.b.c.69. This is the destination address that the internet host originally sent the request to, and it should be the source address of the reply when BINAT is functioning correctly. So in brief:

Expected:

Internet Host sends a request
source: 1.2.3.4
destination: a.b.c.69 -> NAT -> 10.7.24.60
LAN Host sends a reply
source: 10.7.24.60 -> NAT -> a.b.c.69
destination: 1.2.3.4

Observed:

Internet Host sends a request
source: 1.2.3.4
destination: a.b.c.69 -> NAT -> 10.7.24.60
LAN Host sends a reply
source: 10.7.24.60 -> no NAT -> 10.7.24.60
destination: 1.2.3.4

The problem here is that pfSense is forwarding a packet to its upstream gateway with an rfc1918 address as source, which is non-routable on the public internet and is just dropped by the upstream gateway and the internet host never receives a reply.

edit: As mentioned in a previous post, the exception is when the internet host is on the same subnet as PUB01, ie, a.b.c.64/26. In this case, the reply has its source address rewritten to a.b.c.69 as expected, and the reply is forwarded directly to the internet host as expected. But if the reply is forwarded to the upstream gateway, the source address is not rewritten.

edit: I followed all of the troubleshooting steps here, here, here, here and here. I've tried port forwards and 1:1 NAT. I've tried automatic and manual outbound NAT.

I've been using pfSense many years, and while I still occasionally make mistakes, I've been looking at this problem for many hours now and I'm at the point where I'm really starting to think this is a bug. I hope I'm wrong and somebody will point out where I'm wrong.

viragomann

@clarknova said in Outbound NAT not working for single host (multi-WAN):

Expected:
Internet Host sends a request
source: 1.2.3.4
destination: a.b.c.69 -> NAT -> 10.7.24.60
LAN Host sends a reply
source: 10.7.24.60 -> NAT -> a.b.c.69
destination: 1.2.3.4

Replying with the origin destination IP is the default behavior when natting the traffic to an inside IP. There is no need to add special NAT rules for that.

However, 2.5.1 has a bug which let pfSense blow out response packets to the default gateway, regaredless which interface the request was coming in. So possibly you're affected and your problem comes from there.

Saw a post recently which said that 2.5.2 is already available: https://forum.netgate.com/topic/164926/pfsense-ce-2-5-2-release-now-available?_=1625666893002
Maybe that will solve your issue.

clarknova

@viragomann

Thank you, I think you've pointed me in the right direction. The release of 2.5.2 could not have come at a better time! Well, last week before I upgraded to 2.5.1 would have been better, but I'll take today.

https://redmine.pfsense.org/issues/11805