Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Outbound NAT not working for single host (multi-WAN)

    Scheduled Pinned Locked Moved NAT
    6 Posts 2 Posters 511 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      clarknova
      last edited by clarknova

      pfsense 2.5.1

      This firewall is dual-WAN with virtual IPs on both WAN interfaces.

       PUB03 (wan)     -> vmx1       -> v4: x.y.z.43/26
       LAN (lan)       -> vmx0       -> v4: 10.7.24.254/24
       PUB01 (opt1)    -> vmx2       -> v4: a.b.c.86/26
      
      ifconfig vmx2|grep a
      	inet a.b.c.86 netmask 0xffffffc0 broadcast a.b.c.127
      	inet a.b.c.116 netmask 0xffffffc0 broadcast a.b.c.127
      	inet a.b.c.69 netmask 0xffffffc0 broadcast a.b.c.127
      	inet a.b.c.78 netmask 0xffffffc0 broadcast a.b.c.127
      	inet a.b.c.115 netmask 0xffffffc0 broadcast a.b.c.127
      

      Multiple Port Forward rules are configured on the various public addresses and multiple outbound NAT rules are configured. For the most part, everything works as expected, but there is one exception that I haven't been able to figure out. I've attached screenshots of the port forward and outbound NAT rules that pertain to this host. For reasons I haven't been able to nail down, when I telnet to a.b.c.69 on port 46300, the reply exits the PUB01 interface with the local IP address intact like this:

      tcpdump -nivmx2 port 46300
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on vmx2, link-type EN10MB (Ethernet), capture size 262144 bytes
      16:51:47.883040 IP q.r.s.t.48204 > a.b.c.69.46300: Flags [S], seq 2523553336, win 64240, options [mss 1460,sackOK,TS val 680265598 ecr 0,nop,wscale 7], length 0
      16:51:47.886255 IP 10.7.24.60.46300 > q.r.s.t.48204: Flags [S.], seq 3792540171, ack 2523553337, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 699027 ecr 680265598], length 0
      

      What am I doing wrong here?port_forward.png aon.png

      edit: One further note, if I 'telnet a.b.c.69 46300' from a host on the same subnet as the PUB01 interface, the reply gets its source address rewritten as expected and the connection succeeds.

      db

      C 1 Reply Last reply Reply Quote 0
      • C
        clarknova @clarknova
        last edited by

        I disabled the port forward and outbound NAT rules for 10.7.24.60 and enabled 1:1 NAT without changing the firewall rules, but I'm still seeing the same thing, which is non-NAT packets from this host on the PUB01 interface. Here is a dump of my NAT rules:

        pfctl -sn 
        no nat proto carp all
        nat-anchor "natearly/*" all
        nat-anchor "natrules/*" all
        nat on vmx2 inet from 10.7.24.101 to any -> a.b.c.116 port 1024:65535
        nat on vmx2 inet from 10.7.24.102 to any -> a.b.c.115 port 1024:65535
        nat on vmx2 inet from 10.7.24.125 to any -> a.b.c.86 port 1024:65535
        nat on vmx1 inet from 10.7.24.126 to any -> x.y.z.28 port 1024:65535
        nat on vmx2 inet from 10.7.24.127 to any -> a.b.c.78 port 1024:65535
        nat on vmx1 inet from 10.7.24.135 to any -> x.y.z.31 port 1024:65535
        nat on vmx1 inet from 10.7.24.0/24 to any -> x.y.z.43 port 1024:65535
        nat on vmx1 inet from (self) to any port = isakmp -> x.y.z.43 static-port
        nat on vmx1 inet6 from (self) to any port = isakmp -> (vmx1) round-robin static-port
        nat on vmx2 inet from (self) to any port = isakmp -> a.b.c.86 static-port
        nat on vmx2 inet6 from (self) to any port = isakmp -> (vmx2) round-robin static-port
        no rdr proto carp all
        rdr-anchor "tftp-proxy/*" all
        rdr on vmx2 inet proto tcp from any to a.b.c.116 port = 46300 -> 10.7.24.101
        rdr on vmx2 inet proto tcp from any to a.b.c.116 port = 51502 -> 10.7.24.101
        rdr on vmx2 inet proto tcp from any to a.b.c.116 port 7137:7140 -> 10.7.24.101
        rdr on vmx1 inet proto tcp from any to x.y.z.43 port 7137:7140 -> 10.7.24.100
        rdr on vmx1 inet proto tcp from any to x.y.z.43 port = 46300 -> 10.7.24.100
        rdr on vmx1 inet proto tcp from any to x.y.z.43 port = 51502 -> 10.7.24.100
        rdr pass on vmx2 inet proto tcp from any to a.b.c.86 port = http -> 10.7.24.125
        rdr pass on vmx2 inet proto tcp from any to a.b.c.86 port = https -> 10.7.24.125
        rdr-anchor "miniupnpd" all
        binat on vmx2 inet from 10.7.24.101 to any -> a.b.c.116
        binat on vmx1 inet from 10.7.24.126 to any -> x.y.z.28
        binat on vmx2 inet from 10.7.24.60 to any -> a.b.c.69
        binat on vmx2 inet from 10.7.24.127 to any -> a.b.c.78
        binat on vmx2 inet from 10.7.24.102 to any -> a.b.c.115
        binat on vmx1 inet from 10.7.24.135 to any -> x.y.z.31
        

        I don't see any problem here, but maybe somebody else does.

        db

        V 1 Reply Last reply Reply Quote 0
        • V
          viragomann @clarknova
          last edited by

          @clarknova
          Not really clear what you're trying to achieve here.
          10.7.24.60 is in you LAN. When you forward packets from PUB01 to it, the packets won't go out on PUB01 but on LAN. So the outbound NAT rule on PUB01 for the sourece 10.7.24.60/32 is quite useless.

          C 1 Reply Last reply Reply Quote 0
          • C
            clarknova @viragomann
            last edited by clarknova

            @viragomann, PUB01 is on the internet. Forwarding packets from PUB01 to 10.7.24.60 is working fine, ie, the destination address is rewritten to 10.7.24.60 as expected, but when 10.7.24.60 replies, the reply is forwarded back out of PUB01, which is also expected. What is not expected is that the source address of the reply is not rewritten to the public virtual IP of PUB01, a.b.c.69. This is the destination address that the internet host originally sent the request to, and it should be the source address of the reply when BINAT is functioning correctly. So in brief:

            Expected:

            1. Internet Host sends a request
              source: 1.2.3.4
              destination: a.b.c.69 -> NAT -> 10.7.24.60
            2. LAN Host sends a reply
              source: 10.7.24.60 -> NAT -> a.b.c.69
              destination: 1.2.3.4

            Observed:

            1. Internet Host sends a request
              source: 1.2.3.4
              destination: a.b.c.69 -> NAT -> 10.7.24.60
            2. LAN Host sends a reply
              source: 10.7.24.60 -> no NAT -> 10.7.24.60
              destination: 1.2.3.4

            The problem here is that pfSense is forwarding a packet to its upstream gateway with an rfc1918 address as source, which is non-routable on the public internet and is just dropped by the upstream gateway and the internet host never receives a reply.

            edit: As mentioned in a previous post, the exception is when the internet host is on the same subnet as PUB01, ie, a.b.c.64/26. In this case, the reply has its source address rewritten to a.b.c.69 as expected, and the reply is forwarded directly to the internet host as expected. But if the reply is forwarded to the upstream gateway, the source address is not rewritten.

            edit: I followed all of the troubleshooting steps here, here, here, here and here. I've tried port forwards and 1:1 NAT. I've tried automatic and manual outbound NAT.

            I've been using pfSense many years, and while I still occasionally make mistakes, I've been looking at this problem for many hours now and I'm at the point where I'm really starting to think this is a bug. I hope I'm wrong and somebody will point out where I'm wrong.

            db

            V 1 Reply Last reply Reply Quote 0
            • V
              viragomann @clarknova
              last edited by

              @clarknova said in Outbound NAT not working for single host (multi-WAN):

              Expected:
              Internet Host sends a request
              source: 1.2.3.4
              destination: a.b.c.69 -> NAT -> 10.7.24.60
              LAN Host sends a reply
              source: 10.7.24.60 -> NAT -> a.b.c.69
              destination: 1.2.3.4

              Replying with the origin destination IP is the default behavior when natting the traffic to an inside IP. There is no need to add special NAT rules for that.

              However, 2.5.1 has a bug which let pfSense blow out response packets to the default gateway, regaredless which interface the request was coming in. So possibly you're affected and your problem comes from there.

              Saw a post recently which said that 2.5.2 is already available: https://forum.netgate.com/topic/164926/pfsense-ce-2-5-2-release-now-available?_=1625666893002
              Maybe that will solve your issue.

              C 1 Reply Last reply Reply Quote 1
              • C
                clarknova @viragomann
                last edited by

                @viragomann

                Thank you, I think you've pointed me in the right direction. The release of 2.5.2 could not have come at a better time! Well, last week before I upgraded to 2.5.1 would have been better, but I'll take today.

                https://redmine.pfsense.org/issues/11805

                db

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.