Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Impossible to use shared CARP WAN IP for outbound traffic

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    18 Posts 3 Posters 4.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      I'd stop using traceroute to debug DNS problems and use drill/dig instead.

      If you change outbound NAT to the CARP WAN IP and can still ping out from LAN by IP address and, say, telnet out to mail servers by IP address on port 25/587, but cannot resolve names, you have a DNS problem. Use DNS tools to find the failure, not routing/pinging tools.

      Examining the state table should prove the above pings and telnets are using the CARP VIP.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • R
        rmelilloii
        last edited by

        Good point, I'll do a new test and look at the States, it should make things easier to diagnostic, but when using WAN CARP IP for Outbound I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

        Thank you!

        1 Reply Last reply Reply Quote 0
        • DerelictD
          Derelict LAYER 8 Netgate
          last edited by

          I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

          No idea what you're saying here. Sorry.

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          1 Reply Last reply Reply Quote 0
          • R
            rmelilloii
            last edited by

            Well, when I set the CARP WAN IP on my NAT outbound, from my servers I can ping an address by IP and by name. That is why I don't think that it would be a DNS issue.

            Ex.: ping google.com - OK
                  ping 8.8.8.8 - OK

            I lose the "ability" to open a site on browser, access an external FTP, access an external RDP.

            Thank you again, some times I'm not able to formulate well what I'm thinking :)

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              Are you using squid?

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • R
                rmelilloii
                last edited by

                No… Just CARP for redundancy, NAT for port redirection to my internal servers/Load Balancer.

                It is a really strange "behaviour" I think.

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Problem is doing what you're trying to do works just fine. Are you policy routing on your LAN rules?

                  There's a pretty good walk through on setting up CARP in the book. Do you have access to that?

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • R
                    rmelilloii
                    last edited by

                    Hello, good morning!

                    Yes, I agree, based on the documentation (I have the previous version, but is fine I think) and on others environments, everything is OK.

                    The HA work perfectly, all inbound rules using CARP WAN IP, the only thing not good is when trying to set the outbound rules to use CARP WAN IP.

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      It works fine. You'll have to provide more information like what http proxy settings you have. Sounds like it might be firewall rules upstream or something.

                      try "telnet www.host.com 80" from a LAN host. what does that do?

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • R
                        rmelilloii
                        last edited by

                        No proxy…

                        About the telnet

                        With WAN address is ok.
                        CARP WAN IP: C:\Windows\system32>telnet www.google.com 80
                        Connecting To www.google.com...Could not open connection to the host, on port 80
                        : Connect failed

                        I even re-applied the settings to my vSwitches:
                        Enable promiscuous mode on the vSwitch
                        Enable "MAC Address changes"
                        Enable "Forged transmits"

                        But nothing has changed, and I have all incoming traffic using the CARP WAN IP, so I don't think that is related to it.

                        **For test purpose I just added a new IP ALIAS: .248/28, I set it on the outbound rule and it is working. So, maybe we have a BUG with CARP IP for Outbound?

                        WAN ADDRESS

                        Outbound NAT rules (manual)

                        nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                        nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                        nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                        NEW TEST IP (ALIAS)

                        Outbound NAT rules (manual)

                        nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                        nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 192.168.100.0/24 to any -> XXX.XXX.50.248/32 port 1024:65535 
                        nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                        CARP IP (CARP)

                        Outbound NAT rules (manual)

                        nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                        nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.66/32 port 1024:65535 
                        nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                        nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                        WAN ADDRESS : Internet OK - IP: .68
                        NEW TEST IP: Internet OK - IP: .248
                        CARP IP: No Internet

                        1 Reply Last reply Reply Quote 0
                        • dotdashD
                          dotdash
                          last edited by

                          There is no bug. Using the CARP for outbound is standard in this configuration and many are using it with no problems. Perhaps an issue with the provider equipment or your settings.

                          1 Reply Last reply Reply Quote 0
                          • R
                            rmelilloii
                            last edited by

                            Hello, good afternoon!

                            Thank you for your comment and I really think that the config is fine, so if you or someone could/want to check, here is my conf file:

                            • Is really strange the fact of the inbound is working and just the outbound not.

                            Thank you!

                            set optimization normal
                            set limit states 98000
                            set limit src-nodes 98000

                            #System aliases

                            loopback = "{ lo0 }"
                            WAN = "{ em0 }"
                            LAN = "{ em1 }"
                            SYNC = "{ em2 }"

                            #SSH Lockout Table
                            table <sshlockout>persist
                            table <webconfiguratorlockout>persist
                            #Snort tables
                            table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
                            table <bogonsv6>persist file "/etc/bogonsv6"
                            table <negate_networks># User Aliases

                            Gateways

                            GWGW_WAN = " route-to ( em0 XX.XX.143.78 ) "

                            set loginterface em1

                            set skip on pfsync0

                            scrub on $WAN all    fragment reassemble
                            scrub on $LAN all    fragment reassemble
                            scrub on $SYNC all    fragment reassemble

                            no nat proto carp
                            no rdr proto carp
                            nat-anchor "natearly/"
                            nat-anchor "natrules/                            "

                            Outbound NAT rules (manual)

                            nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                            nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                            nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                            nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                            nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                            nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                            Load balancing anchor

                            rdr-anchor "relayd/*"

                            TFTP proxy

                            rdr-anchor "tftp-proxy/*"

                            NAT Inbound Redirects

                            rdr on em0 proto tcp from any to XX.XX.143.66 port 13389 -> 192.168.100.241 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 23389 -> 192.168.100.242 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 33389 -> 192.168.100.244 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 43389 -> 192.168.100.245 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 53389 -> 192.168.100.247 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 63389 -> 192.168.100.248 port 3389
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 80 -> 192.168.100.10
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 180 -> 192.168.100.241 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 280 -> 192.168.100.242 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 380 -> 192.168.100.244 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 480 -> 192.168.100.245 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 580 -> 192.168.100.247 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 680 -> 192.168.100.248 port 80
                            rdr on em0 proto tcp from any to XX.XX.143.66 port 443 -> 192.168.100.10

                            UPnPd rdr anchor

                            rdr-anchor "miniupnpd"

                            anchor "relayd/"
                            anchor "openvpn/                            "
                            anchor "ipsec/*"

                            block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,

                            and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but

                            route-to can override that, causing problems such as in redmine #2073

                            block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
                            block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
                            #–-------------------------------------------------------------------------

                            default deny rules

                            #---------------------------------------------------------------------------
                            block in log inet all tracker 1000000103 label "Default deny rule IPv4"
                            block out log inet all tracker 1000000104 label "Default deny rule IPv4"
                            block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
                            block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

                            IPv6 ICMP is not auxilary, it is required for operation

                            See man icmp6(4)

                            1    unreach        Destination unreachable

                            2    toobig          Packet too big

                            128  echoreq        Echo service request

                            129  echorep        Echo service reply

                            133  routersol      Router solicitation

                            134  routeradv      Router advertisement

                            135  neighbrsol      Neighbor solicitation

                            136  neighbradv      Neighbor advertisement

                            pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

                            Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

                            pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
                            pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
                            pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
                            pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
                            pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state

                            We use the mighty pf, we cannot be fooled.

                            block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
                            block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
                            block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
                            block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"

                            Snort package

                            block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
                            block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
                            block in log quick proto carp from (self) to any tracker 1000000201
                            pass  quick proto carp tracker 1000000202

                            SSH lockout

                            block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"

                            webConfigurator lockout

                            block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
                            block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

                            block bogon networks (IPv4)

                            http://www.cymru.com/Documents/bogon-bn-nonagg.txt

                            block in log quick on $WAN from <bogons>to any tracker 1000001561 label "block bogon IPv4 networks from WAN"

                            block bogon networks (IPv6)

                            http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

                            block in log quick on $WAN from <bogonsv6>to any tracker 1000001562 label "block bogon IPv6 networks from WAN"
                            antispoof log for $WAN tracker 1000001570

                            block anything from private networks on interfaces with the option set

                            block in log quick on $WAN from 10.0.0.0/8 to any tracker 1000001581 label "Block private networks from WAN block 10/8"
                            block in log quick on $WAN from 127.0.0.0/8 to any tracker 1000001582 label "Block private networks from WAN block 127/8"
                            block in log quick on $WAN from 172.16.0.0/12 to any tracker 1000001583 label "Block private networks from WAN block 172.16/12"
                            block in log quick on $WAN from 192.168.0.0/16 to any tracker 1000001584 label "Block private networks from WAN block 192.168/16"
                            block in log quick on $WAN from fc00::/7 to any tracker 1000001585 label "Block ULA networks from WAN block fc00::/7"
                            antispoof log for $LAN tracker 1000002620
                            antispoof log for $SYNC tracker 1000003670

                            loopback

                            pass in  on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
                            pass out  on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
                            pass in  on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
                            pass out  on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"

                            let out anything from the firewall host itself and decrypted IPsec traffic

                            pass out  inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
                            pass out  inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"
                            pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.68 to !XX.XX.143.64/28 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
                            pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.66 to !XX.XX.143.64/28 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"

                            make sure the user cannot lock himself out of the webConfigurator or SSH

                            pass in  quick on em1 proto tcp from any to (em1) port { 80 } tracker 1000004121 keep state label "anti-lockout rule"

                            User-defined rules follow

                            anchor "userrules/*"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 443 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 ) inet proto icmp  from any to any tracker 1450788722 keep state  label "USER_RULE: PING"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS02"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS02"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS02"
                            pass  in  quick  on $LAN inet proto icmp  from any to any tracker 1450786062 keep state  label "USER_RULE: PING"
                            pass  in  quick  on $LAN inet from 192.168.100.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"

                            at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

                            pass  in  quick  on $SYNC inet from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                            pass  in  quick  on $SYNC inet6 from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS02"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS02"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS01"
                            pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS02"

                            VPN Rules

                            anchor "tftp-proxy/*"</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>

                            1 Reply Last reply Reply Quote 0
                            • R
                              rmelilloii
                              last edited by

                              So, thanks to everyone!!

                              Was an issue related to the specific IP.

                              In love with Pfsense again :)

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.