Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2 WAN, 1:1 NAT, Outgoing not working (Solved)

    Scheduled Pinned Locked Moved
    NAT
    2
    14
    6.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      preid
      last edited by

      Hi guys,

      I have the following setup on pfSense 2.3.2:

      WAN1: 10.10.100.241/28
      WAN2: 10.10.200.161/28
      LAN PC: 192.168.230.8/24

      I am trying to tie the LAN PC to the WAN2 IP of 10.10.200.172 for both incoming and outgoing traffic.

      I did set up a 1:1 NAT rule that worked for incoming, but outgoing still goes out the 10.10.100.241 WAN interface.

      I then tried to set up an Outgoing NAT rule, and that only works if I tied it to WAN1 (even though the VIP is have configured is on WAN2), and it causes all kinds of weird problems with traffic flow when I do this (stuff like getting a single ping through, then nothing after that, and the PC can no longer reach anything on either of the WAN interfaces of the pfSense box).

      Obviously, I have this screwed up somehow, or I'm doing this entirely wrong. I'm assuming there is a better way than 1:1 NAT for this?

      As I have it set up right now:

      1:1 NAT:

      • Interface: WAN2
      • External Subnet IP: 10.10.200.172
      • Internal IP: Single Host, 192.168.230.8
      • Destination: Any

      Outbound NAT:

      • Interface: WAN1
      • Protocol: Any
      • Source: Network, 192.168.230.8/32
      • Destination: Any
      • Translation Address: 10.10.200 172

      I assume the issue is that I'm having to trap the outbound NAT on WAN1, which of course is then sending traffic out the wrong WAN interface - but if I use WAN2, the traffic is never captured and translated. It seems like I need a way to tell it where to find the traffic (WAN1), then to translate it and send it out WAN2, but there doesn't seem to be a way to do this. Like I say, though, maybe I'm going about this all wrong and there's a better way to assign a WAN IP to a LAN IP that I'm missing?

      I've done this kind of thing with SonicWALLs in the past, but I can't see to get pfSense to do the job. Any suggestions would be great.

      Thanks!

      1 Reply Last reply Reply Quote 0
      • P
        preid
        last edited by

        Doing some checking into this, it appears everything works, so long as the master CARP node responds to ARP request from the outside node (sending 00:0c:29:46:bb:2f).

        What's happening, though, is that sometimes the backup CARP node is responding to ARP first (sending 00:50:56:b9:99:63), and then my outside machines can't reach the inside machine through the Virtual VIP for the 1:1 NAT setup, since traffic is now going to the backup node - thus I end up with this "sometimes it works, sometimes it doesn't" situation, depending on which node replied to the ARP request first.

        Why is the slave CARP node responding to ARP for a virtual IP when it can't route the traffic anyway? Is this a known issue, or am I screwing something up that's causing this?

        I haven't checked, but I wouldn't be surprised if this is also happening with the CARP VIP's themselves, since they seem to work off and on, too for pfSense management from the WAN.

        1 Reply Last reply Reply Quote 0
        • DerelictD
          Derelict LAYER 8 Netgate
          last edited by

          Sounds like you're screwing something up.

          Where is CARP in the mix? Is this an HA cluster? What are the pertinent interface and CARP VIP addresses?

          1:1 NAT does not route traffic. If you want traffic to egress out a specific interface you have to policy route it. While traffic is going outbound out a specific interface as determined by policy routing then the routing table, 1:1 NAT will be consulted and the source address translated accordingly.

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          1 Reply Last reply Reply Quote 0
          • P
            preid
            last edited by

            Hi, Derelict,

            CARP is looking after a dual node cluster. I have 5 physical interfaces - WAN1, WAN2, LAN1, LAN2, and the CARP replication. All have CARP VIPs.

            I set up my 1:1 NAT using an IP Alias VIP in the WAN2 subnet. Maybe it should be another CARP VIP, instead? Maybe that's where I'm screwed up?

            I'll look into the policy route to get my traffic for my WAN2 1:1 NAT to go out the WAN2 interface if it originates from the internal machine. I know that on a SonicWALL you do this via the NAT rules (SonicWALL also wants a loopback rule, so internal machines can reach the 1:1 NAT machine via the WAN IP. Do I need the same sort of thing with pfSense, as well?)

            Thanks for the help, Derelict.

            1 Reply Last reply Reply Quote 0
            • P
              preid
              last edited by

              Ok. My source route totally solved my outgoing traffic for the 1:1 NAT setup, which I'm super happy about.

              Now if I could just get my backup CARP node to stop ARPing for my 1:1 NAT WAN IP, I'd be set. Every time it does that, communications breaks down because traffic goes back to to the backup node, which never gets to my internal PC.

              I'm getting closer, though! Maybe I'll try switching the IP Alias VIP to a CARP VIP, and that'll stop the backup from ARPing for things it can't move traffic for? Or should I be doing something else with CARP in the mix?

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                IP Alias VIPs in a CARP setup should be tied to the CARP VIP. They will swing over to the backup when the CARP VIP does.

                Delete the VIP on the secondary and edit the VIP on the primary changing the interface to the CARP VIP.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • P
                  preid
                  last edited by

                  Ok. I deleted the IP Aliases from my backup node, then switched my IP Aliases on the master node from type IP Alias to type CARP, fed it a password and VHID Group, and saved. I think that is what you were telling me to do. :)

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    Not really. You can keep the IP Alias VIP but you only want to define it on the primary and change the interface on the VIP to a CARP VIP on that interface.

                    The more CARP you have on an interface the more heartbeat traffic there is and the more likely heartbeats will be missed on the secondary. As long as there are not a lot it should be fine but the IP Alias VIPs tacked onto CARP VIPs is a pretty elegant solution.

                    ![Screen Shot 2016-10-03 at 4.05.12 PM.png](/public/imported_attachments/1/Screen Shot 2016-10-03 at 4.05.12 PM.png)
                    ![Screen Shot 2016-10-03 at 4.05.12 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-10-03 at 4.05.12 PM.png_thumb)

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • P
                      preid
                      last edited by

                      Ah ha!

                      The image you attached really made it clear what you meant. I made the change, and things seem to be working pretty well now.

                      The only issue I seem to have left is that my machine on WAN2 that has been 1:1 NAT'd, policy routed out WAN2, and outbound NAT'd to it's 1:1 NAT IP can't talk to anything through my port forwards on WAN1. It can ping stuff, but it can't connect with anything TCP.

                      I'm thinking I need some kind of loopback route rule somewhere?

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        Hmm. That should work OK.

                        https://doc.pfsense.org/index.php/Port_Forward_Troubleshooting

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • P
                          preid
                          last edited by

                          I would have thought so, too, but it doesn't seem to.

                          This is what I have:

                          WAN1: 10.10.100.241/28
                          WAN2: 10.10.200.161/28
                          LAN PC 1: 192.168.230.190/24
                          LAN PC 2: 192.168.230.8/24

                          1:1 NAT:

                          • Interface: WAN1

                          • External IP: 10.10.100.244

                          • Internal IP: 192.168.230.190

                          • Interface: WAN2

                          • External IP: 10.10.200.172

                          • Internal IP: Single host, 192.168.230.8

                          Outbound NAT (in Hybrid Outbound NAT mode):

                          • Interface: WAN1

                          • Source: Any

                          • Destination: Any

                          • NAT Address: 10.10.100.244

                          • Interface: WAN2

                          • Source: Any

                          • Destination: Any

                          • NAT Address: 10.10.200.172

                          VIPs:

                          • Type: IP Alias

                          • Interface: 10.10.100.241 (WAN1 IP)

                          • Address: 10.10.100.244/32

                          • Type: IP Alias

                          • Interface: 10.10.200.161 (WAN2 IP)

                          • Address: 10.10.200.172/32

                          Firewall (Policy route only for WAN2 - WAN1 didn't need it, since it's the default gateway):

                          • Interface: LAN

                          • Protocol: Any

                          • Source: Single host, 192.168.230.8

                          • Gateway: OPT3GW - 10.10.200.174 - WAN2 Gateway

                          • Interface: WAN1

                          • Protocol: HTTP

                          • Source: Any

                          • Destination: Single host, 192.168.230.190

                          • Interface: WAN1

                          • Protocol: ICMP

                          • Source: Any

                          • Destination: Single host, 192.168.230.190

                          • Interface: WAN2

                          • Protocol: HTTP

                          • Source: Any

                          • Destination: Single host, 192.168.230.8

                          • Interface: WAN2

                          • Protocol: ICMP

                          • Source: Any

                          • Destination: Single host, 192.168.230.8

                          Results:

                          • 192.168.230.8 can ping 10.10.100.244
                          • 192.168.230.190 can ping 10.10.200.172 (as can a couple of other internal hosts, but many internal hosts get ping timeouts, even though they all use the same LAN gateway - 192.168.230.1, which is the pfSense LAN CARP VIP)
                          • Neither LAN host can connect via HTTP to the other via their IP Aliases
                          • Both hosts can ping and connect to the pfSense management interface on the opposite WAN (both the VIP and the real IPs of the pfSense boxes)
                          • Both LAN hosts can be reached via ICMP and HTTP from the WAN by external hosts

                          Looking at the state tables, it appears that traffic between WAN interfaces should be handled internally - the traffic doesn't appear to be heading out to the external ISP gateways at all, which is what I would expect. When pinging from 192.168.230.8 to 10.10.100.244, for example, I can see a state of:

                          LAN - ICMP - 192.168.230.8:4 -> 192.168.230.190:4 (10.10.100.244:4) - 0:0 - 4 / 0

                          But if I try to connect via HTTP, I see a state of:

                          LAN - TCP - 192.168.230.8:17123 -> 192.168.230.190:80 (10.10.100.244:80) - CLOSED:SYN_SENT - 3 / 0

                          Yet I have no problem connecting via HTTP directly using the private IPs (192.168.230.8 -> 192.168.230.190).

                          192.168.230.8 can't connect through any port forwards, in fact - not just the ones with 1:1 NAT and stuff configured. Even connecting to one my basic port forwards in to another internal host through the WAN1 CARP VIP fails, though they work fine from external hosts.

                          Somehow, my traffic is dropped on it's way from the LAN through WAN1 and back into the LAN. This happens on other routers I've worked with, too, and required a "loopback rule" to get working to translate the LAN traffic to appear to be traffic coming into the WAN2 interface, where all the normal firewall rules then apply. It seems like pfSense needs something similar somewhere, perhaps?

                          1 Reply Last reply Reply Quote 0
                          • DerelictD
                            Derelict LAYER 8 Netgate
                            last edited by

                            Source any on outbound NAT is almost never right. You will be NATting things you don't want to NAT, like traffic sourced from the firewall. Don't be lazy here. Define rules for the inside networks you want to NAT for.

                            If you are testing all of this from inside expecting to ping pong from the inside to the WAN address and back inside you need to look at NAT reflection.

                            Ugly hack (it is on the other firewalls too) but you seem to be insistent on it. Split DNS is another (better) option.

                            https://doc.pfsense.org/index.php/Why_can%27t_I_access_forwarded_ports_on_my_WAN_IP_from_my_LAN/OPTx_networks

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • P
                              preid
                              last edited by

                              That was my mistake when transcribing my setup. I did, indeed, have a source entered for my outbound NAT rules (the LAN IP of the related host/32).

                              I wouldn't use such a hack, except I know some of the machines that will be coming into the internal network are from a data center where they were behind a transparent filtering bridge and communicated with each other using their WAN IPs, since they didn't have any LAN IPs at the time. Over time I can clean this up and kill the hack, but until that happens, the external IPs need to work from the internal hosts to avoid a mess.

                              Turning on the ugly hack does allow me to reach my 192.168.230.8 (10.10.200.172) machine from my other LAN machines using the 10.10.200.172 IP, but oddly it doesn't seem to allow my 192.168.230.8 machine to reach my 10.10.100.244 machine on WAN1. Even ping stopped working. WAN1 -> WAN2 is all good. WAN2 -> WAN1, not so much. Weird.

                              I see in the state table:

                              LAN - ICMP - 192.168.230.8:1 -> 192.168.230.190:1 (10.10.100.244:1) - 0:0 - 3 / 0
                              WAN2 - ICMP - 10.10.200.172:1 (192.168.230.8:1) -> 192.168.230.190:1 - 0:0 - 3 / 0

                              By the way, thank you so much for all your help. I know the set up has a lot of moving parts, and I appreciate your assistance in nailing down the issues I've run into.

                              1 Reply Last reply Reply Quote 0
                              • P
                                preid
                                last edited by

                                I figured it out. My 1:1 NAT rule for 192.168.230.8 had NAT Reflection set to Enabled, whereas the 1:1 NAT rule for 192.168.230.190 has NAT Reflect set to System Default. As soon as I switched it to Enabled, things started working. :)

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post