Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Impossible to use shared CARP WAN IP for outbound traffic

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    18 Posts 3 Posters 4.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rmelilloii
      last edited by

      Well, when I set the CARP WAN IP on my NAT outbound, from my servers I can ping an address by IP and by name. That is why I don't think that it would be a DNS issue.

      Ex.: ping google.com - OK
            ping 8.8.8.8 - OK

      I lose the "ability" to open a site on browser, access an external FTP, access an external RDP.

      Thank you again, some times I'm not able to formulate well what I'm thinking :)

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        Are you using squid?

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • R
          rmelilloii
          last edited by

          No… Just CARP for redundancy, NAT for port redirection to my internal servers/Load Balancer.

          It is a really strange "behaviour" I think.

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            Problem is doing what you're trying to do works just fine. Are you policy routing on your LAN rules?

            There's a pretty good walk through on setting up CARP in the book. Do you have access to that?

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • R
              rmelilloii
              last edited by

              Hello, good morning!

              Yes, I agree, based on the documentation (I have the previous version, but is fine I think) and on others environments, everything is OK.

              The HA work perfectly, all inbound rules using CARP WAN IP, the only thing not good is when trying to set the outbound rules to use CARP WAN IP.

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                It works fine. You'll have to provide more information like what http proxy settings you have. Sounds like it might be firewall rules upstream or something.

                try "telnet www.host.com 80" from a LAN host. what does that do?

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • R
                  rmelilloii
                  last edited by

                  No proxy…

                  About the telnet

                  With WAN address is ok.
                  CARP WAN IP: C:\Windows\system32>telnet www.google.com 80
                  Connecting To www.google.com...Could not open connection to the host, on port 80
                  : Connect failed

                  I even re-applied the settings to my vSwitches:
                  Enable promiscuous mode on the vSwitch
                  Enable "MAC Address changes"
                  Enable "Forged transmits"

                  But nothing has changed, and I have all incoming traffic using the CARP WAN IP, so I don't think that is related to it.

                  **For test purpose I just added a new IP ALIAS: .248/28, I set it on the outbound rule and it is working. So, maybe we have a BUG with CARP IP for Outbound?

                  WAN ADDRESS

                  Outbound NAT rules (manual)

                  nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                  nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                  nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                  NEW TEST IP (ALIAS)

                  Outbound NAT rules (manual)

                  nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                  nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 192.168.100.0/24 to any -> XXX.XXX.50.248/32 port 1024:65535 
                  nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                  CARP IP (CARP)

                  Outbound NAT rules (manual)

                  nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                  nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.66/32 port 1024:65535 
                  nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                  nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                  WAN ADDRESS : Internet OK - IP: .68
                  NEW TEST IP: Internet OK - IP: .248
                  CARP IP: No Internet

                  1 Reply Last reply Reply Quote 0
                  • dotdashD
                    dotdash
                    last edited by

                    There is no bug. Using the CARP for outbound is standard in this configuration and many are using it with no problems. Perhaps an issue with the provider equipment or your settings.

                    1 Reply Last reply Reply Quote 0
                    • R
                      rmelilloii
                      last edited by

                      Hello, good afternoon!

                      Thank you for your comment and I really think that the config is fine, so if you or someone could/want to check, here is my conf file:

                      • Is really strange the fact of the inbound is working and just the outbound not.

                      Thank you!

                      set optimization normal
                      set limit states 98000
                      set limit src-nodes 98000

                      #System aliases

                      loopback = "{ lo0 }"
                      WAN = "{ em0 }"
                      LAN = "{ em1 }"
                      SYNC = "{ em2 }"

                      #SSH Lockout Table
                      table <sshlockout>persist
                      table <webconfiguratorlockout>persist
                      #Snort tables
                      table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
                      table <bogonsv6>persist file "/etc/bogonsv6"
                      table <negate_networks># User Aliases

                      Gateways

                      GWGW_WAN = " route-to ( em0 XX.XX.143.78 ) "

                      set loginterface em1

                      set skip on pfsync0

                      scrub on $WAN all    fragment reassemble
                      scrub on $LAN all    fragment reassemble
                      scrub on $SYNC all    fragment reassemble

                      no nat proto carp
                      no rdr proto carp
                      nat-anchor "natearly/"
                      nat-anchor "natrules/                      "

                      Outbound NAT rules (manual)

                      nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                      nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                      nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                      nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                      nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                      nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                      Load balancing anchor

                      rdr-anchor "relayd/*"

                      TFTP proxy

                      rdr-anchor "tftp-proxy/*"

                      NAT Inbound Redirects

                      rdr on em0 proto tcp from any to XX.XX.143.66 port 13389 -> 192.168.100.241 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 23389 -> 192.168.100.242 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 33389 -> 192.168.100.244 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 43389 -> 192.168.100.245 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 53389 -> 192.168.100.247 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 63389 -> 192.168.100.248 port 3389
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 80 -> 192.168.100.10
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 180 -> 192.168.100.241 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 280 -> 192.168.100.242 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 380 -> 192.168.100.244 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 480 -> 192.168.100.245 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 580 -> 192.168.100.247 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 680 -> 192.168.100.248 port 80
                      rdr on em0 proto tcp from any to XX.XX.143.66 port 443 -> 192.168.100.10

                      UPnPd rdr anchor

                      rdr-anchor "miniupnpd"

                      anchor "relayd/"
                      anchor "openvpn/                      "
                      anchor "ipsec/*"

                      block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,

                      and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but

                      route-to can override that, causing problems such as in redmine #2073

                      block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
                      block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
                      #–-------------------------------------------------------------------------

                      default deny rules

                      #---------------------------------------------------------------------------
                      block in log inet all tracker 1000000103 label "Default deny rule IPv4"
                      block out log inet all tracker 1000000104 label "Default deny rule IPv4"
                      block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
                      block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

                      IPv6 ICMP is not auxilary, it is required for operation

                      See man icmp6(4)

                      1    unreach        Destination unreachable

                      2    toobig          Packet too big

                      128  echoreq        Echo service request

                      129  echorep        Echo service reply

                      133  routersol      Router solicitation

                      134  routeradv      Router advertisement

                      135  neighbrsol      Neighbor solicitation

                      136  neighbradv      Neighbor advertisement

                      pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

                      Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

                      pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
                      pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
                      pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
                      pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
                      pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state

                      We use the mighty pf, we cannot be fooled.

                      block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
                      block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
                      block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
                      block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"

                      Snort package

                      block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
                      block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
                      block in log quick proto carp from (self) to any tracker 1000000201
                      pass  quick proto carp tracker 1000000202

                      SSH lockout

                      block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"

                      webConfigurator lockout

                      block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
                      block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

                      block bogon networks (IPv4)

                      http://www.cymru.com/Documents/bogon-bn-nonagg.txt

                      block in log quick on $WAN from <bogons>to any tracker 1000001561 label "block bogon IPv4 networks from WAN"

                      block bogon networks (IPv6)

                      http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

                      block in log quick on $WAN from <bogonsv6>to any tracker 1000001562 label "block bogon IPv6 networks from WAN"
                      antispoof log for $WAN tracker 1000001570

                      block anything from private networks on interfaces with the option set

                      block in log quick on $WAN from 10.0.0.0/8 to any tracker 1000001581 label "Block private networks from WAN block 10/8"
                      block in log quick on $WAN from 127.0.0.0/8 to any tracker 1000001582 label "Block private networks from WAN block 127/8"
                      block in log quick on $WAN from 172.16.0.0/12 to any tracker 1000001583 label "Block private networks from WAN block 172.16/12"
                      block in log quick on $WAN from 192.168.0.0/16 to any tracker 1000001584 label "Block private networks from WAN block 192.168/16"
                      block in log quick on $WAN from fc00::/7 to any tracker 1000001585 label "Block ULA networks from WAN block fc00::/7"
                      antispoof log for $LAN tracker 1000002620
                      antispoof log for $SYNC tracker 1000003670

                      loopback

                      pass in  on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
                      pass out  on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
                      pass in  on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
                      pass out  on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"

                      let out anything from the firewall host itself and decrypted IPsec traffic

                      pass out  inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
                      pass out  inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"
                      pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.68 to !XX.XX.143.64/28 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
                      pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.66 to !XX.XX.143.64/28 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"

                      make sure the user cannot lock himself out of the webConfigurator or SSH

                      pass in  quick on em1 proto tcp from any to (em1) port { 80 } tracker 1000004121 keep state label "anti-lockout rule"

                      User-defined rules follow

                      anchor "userrules/*"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 443 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 ) inet proto icmp  from any to any tracker 1450788722 keep state  label "USER_RULE: PING"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS02"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS02"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS02"
                      pass  in  quick  on $LAN inet proto icmp  from any to any tracker 1450786062 keep state  label "USER_RULE: PING"
                      pass  in  quick  on $LAN inet from 192.168.100.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"

                      at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

                      pass  in  quick  on $SYNC inet from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                      pass  in  quick  on $SYNC inet6 from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS02"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS02"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS01"
                      pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS02"

                      VPN Rules

                      anchor "tftp-proxy/*"</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>

                      1 Reply Last reply Reply Quote 0
                      • R
                        rmelilloii
                        last edited by

                        So, thanks to everyone!!

                        Was an issue related to the specific IP.

                        In love with Pfsense again :)

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.