Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Impossible to use shared CARP WAN IP for outbound traffic

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    18 Posts 3 Posters 4.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rmelilloii
      last edited by

      Hello, good morning!

      Thank you for your reply.

      Well, is more strange than that. When I change to the CARP WAN IP, I can ping by name and by IP. But is impossible to open a page on browser, to properly access something outside my network.
      That is why I think that is some sort of bug. Is like all traffic is being blocked.

      *My DNS servers are the one from my ISP and google ones.

      New test:
      Diagnostics: Traceroute

      pf_test_01.jpg
      pf_test_01.jpg_thumb
      pf_test_02.jpg
      pf_test_02.jpg_thumb

      1 Reply Last reply Reply Quote 0
      • dotdashD
        dotdash
        last edited by

        Try using a different IP for the CARP. Did you previously have another device using .66? If so, reboot the CPE. Subnet mask on the CARP VIP matches the interface subnet?

        1 Reply Last reply Reply Quote 0
        • R
          rmelilloii
          last edited by

          Hello! Trying to reboot the CPE I think that is a good idea, but that way I'll lost for some time all communication… But thanks!!

          Yes, subnet for both WAN IP and CARP WAN IP are /28.

          *All the inbound traffic is directed to .66 and working fine.

          I think I'll give up :) (I'm migrating from a Load Balancer Cloud service, now using my own hardware, so I have more control of everything already).

          Thank you again, when I have the solution or things start to work like magic I'll post here.

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            I'd stop using traceroute to debug DNS problems and use drill/dig instead.

            If you change outbound NAT to the CARP WAN IP and can still ping out from LAN by IP address and, say, telnet out to mail servers by IP address on port 25/587, but cannot resolve names, you have a DNS problem. Use DNS tools to find the failure, not routing/pinging tools.

            Examining the state table should prove the above pings and telnets are using the CARP VIP.

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • R
              rmelilloii
              last edited by

              Good point, I'll do a new test and look at the States, it should make things easier to diagnostic, but when using WAN CARP IP for Outbound I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

              Thank you!

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

                No idea what you're saying here. Sorry.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • R
                  rmelilloii
                  last edited by

                  Well, when I set the CARP WAN IP on my NAT outbound, from my servers I can ping an address by IP and by name. That is why I don't think that it would be a DNS issue.

                  Ex.: ping google.com - OK
                        ping 8.8.8.8 - OK

                  I lose the "ability" to open a site on browser, access an external FTP, access an external RDP.

                  Thank you again, some times I'm not able to formulate well what I'm thinking :)

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    Are you using squid?

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • R
                      rmelilloii
                      last edited by

                      No… Just CARP for redundancy, NAT for port redirection to my internal servers/Load Balancer.

                      It is a really strange "behaviour" I think.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        Problem is doing what you're trying to do works just fine. Are you policy routing on your LAN rules?

                        There's a pretty good walk through on setting up CARP in the book. Do you have access to that?

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • R
                          rmelilloii
                          last edited by

                          Hello, good morning!

                          Yes, I agree, based on the documentation (I have the previous version, but is fine I think) and on others environments, everything is OK.

                          The HA work perfectly, all inbound rules using CARP WAN IP, the only thing not good is when trying to set the outbound rules to use CARP WAN IP.

                          1 Reply Last reply Reply Quote 0
                          • DerelictD
                            Derelict LAYER 8 Netgate
                            last edited by

                            It works fine. You'll have to provide more information like what http proxy settings you have. Sounds like it might be firewall rules upstream or something.

                            try "telnet www.host.com 80" from a LAN host. what does that do?

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • R
                              rmelilloii
                              last edited by

                              No proxy…

                              About the telnet

                              With WAN address is ok.
                              CARP WAN IP: C:\Windows\system32>telnet www.google.com 80
                              Connecting To www.google.com...Could not open connection to the host, on port 80
                              : Connect failed

                              I even re-applied the settings to my vSwitches:
                              Enable promiscuous mode on the vSwitch
                              Enable "MAC Address changes"
                              Enable "Forged transmits"

                              But nothing has changed, and I have all incoming traffic using the CARP WAN IP, so I don't think that is related to it.

                              **For test purpose I just added a new IP ALIAS: .248/28, I set it on the outbound rule and it is working. So, maybe we have a BUG with CARP IP for Outbound?

                              WAN ADDRESS

                              Outbound NAT rules (manual)

                              nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                              nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                              nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                              NEW TEST IP (ALIAS)

                              Outbound NAT rules (manual)

                              nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                              nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 192.168.100.0/24 to any -> XXX.XXX.50.248/32 port 1024:65535 
                              nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                              CARP IP (CARP)

                              Outbound NAT rules (manual)

                              nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                              nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.66/32 port 1024:65535 
                              nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                              nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                              WAN ADDRESS : Internet OK - IP: .68
                              NEW TEST IP: Internet OK - IP: .248
                              CARP IP: No Internet

                              1 Reply Last reply Reply Quote 0
                              • dotdashD
                                dotdash
                                last edited by

                                There is no bug. Using the CARP for outbound is standard in this configuration and many are using it with no problems. Perhaps an issue with the provider equipment or your settings.

                                1 Reply Last reply Reply Quote 0
                                • R
                                  rmelilloii
                                  last edited by

                                  Hello, good afternoon!

                                  Thank you for your comment and I really think that the config is fine, so if you or someone could/want to check, here is my conf file:

                                  • Is really strange the fact of the inbound is working and just the outbound not.

                                  Thank you!

                                  set optimization normal
                                  set limit states 98000
                                  set limit src-nodes 98000

                                  #System aliases

                                  loopback = "{ lo0 }"
                                  WAN = "{ em0 }"
                                  LAN = "{ em1 }"
                                  SYNC = "{ em2 }"

                                  #SSH Lockout Table
                                  table <sshlockout>persist
                                  table <webconfiguratorlockout>persist
                                  #Snort tables
                                  table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
                                  table <bogonsv6>persist file "/etc/bogonsv6"
                                  table <negate_networks># User Aliases

                                  Gateways

                                  GWGW_WAN = " route-to ( em0 XX.XX.143.78 ) "

                                  set loginterface em1

                                  set skip on pfsync0

                                  scrub on $WAN all    fragment reassemble
                                  scrub on $LAN all    fragment reassemble
                                  scrub on $SYNC all    fragment reassemble

                                  no nat proto carp
                                  no rdr proto carp
                                  nat-anchor "natearly/"
                                  nat-anchor "natrules/                                  "

                                  Outbound NAT rules (manual)

                                  nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
                                  nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
                                  nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                                  nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
                                  nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
                                  nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

                                  Load balancing anchor

                                  rdr-anchor "relayd/*"

                                  TFTP proxy

                                  rdr-anchor "tftp-proxy/*"

                                  NAT Inbound Redirects

                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 13389 -> 192.168.100.241 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 23389 -> 192.168.100.242 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 33389 -> 192.168.100.244 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 43389 -> 192.168.100.245 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 53389 -> 192.168.100.247 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 63389 -> 192.168.100.248 port 3389
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 80 -> 192.168.100.10
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 180 -> 192.168.100.241 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 280 -> 192.168.100.242 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 380 -> 192.168.100.244 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 480 -> 192.168.100.245 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 580 -> 192.168.100.247 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 680 -> 192.168.100.248 port 80
                                  rdr on em0 proto tcp from any to XX.XX.143.66 port 443 -> 192.168.100.10

                                  UPnPd rdr anchor

                                  rdr-anchor "miniupnpd"

                                  anchor "relayd/"
                                  anchor "openvpn/                                  "
                                  anchor "ipsec/*"

                                  block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,

                                  and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but

                                  route-to can override that, causing problems such as in redmine #2073

                                  block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
                                  block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
                                  #–-------------------------------------------------------------------------

                                  default deny rules

                                  #---------------------------------------------------------------------------
                                  block in log inet all tracker 1000000103 label "Default deny rule IPv4"
                                  block out log inet all tracker 1000000104 label "Default deny rule IPv4"
                                  block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
                                  block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

                                  IPv6 ICMP is not auxilary, it is required for operation

                                  See man icmp6(4)

                                  1    unreach        Destination unreachable

                                  2    toobig          Packet too big

                                  128  echoreq        Echo service request

                                  129  echorep        Echo service reply

                                  133  routersol      Router solicitation

                                  134  routeradv      Router advertisement

                                  135  neighbrsol      Neighbor solicitation

                                  136  neighbradv      Neighbor advertisement

                                  pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

                                  Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

                                  pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
                                  pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
                                  pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
                                  pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
                                  pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state

                                  We use the mighty pf, we cannot be fooled.

                                  block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
                                  block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
                                  block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
                                  block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"

                                  Snort package

                                  block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
                                  block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
                                  block in log quick proto carp from (self) to any tracker 1000000201
                                  pass  quick proto carp tracker 1000000202

                                  SSH lockout

                                  block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"

                                  webConfigurator lockout

                                  block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
                                  block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

                                  block bogon networks (IPv4)

                                  http://www.cymru.com/Documents/bogon-bn-nonagg.txt

                                  block in log quick on $WAN from <bogons>to any tracker 1000001561 label "block bogon IPv4 networks from WAN"

                                  block bogon networks (IPv6)

                                  http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

                                  block in log quick on $WAN from <bogonsv6>to any tracker 1000001562 label "block bogon IPv6 networks from WAN"
                                  antispoof log for $WAN tracker 1000001570

                                  block anything from private networks on interfaces with the option set

                                  block in log quick on $WAN from 10.0.0.0/8 to any tracker 1000001581 label "Block private networks from WAN block 10/8"
                                  block in log quick on $WAN from 127.0.0.0/8 to any tracker 1000001582 label "Block private networks from WAN block 127/8"
                                  block in log quick on $WAN from 172.16.0.0/12 to any tracker 1000001583 label "Block private networks from WAN block 172.16/12"
                                  block in log quick on $WAN from 192.168.0.0/16 to any tracker 1000001584 label "Block private networks from WAN block 192.168/16"
                                  block in log quick on $WAN from fc00::/7 to any tracker 1000001585 label "Block ULA networks from WAN block fc00::/7"
                                  antispoof log for $LAN tracker 1000002620
                                  antispoof log for $SYNC tracker 1000003670

                                  loopback

                                  pass in  on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
                                  pass out  on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
                                  pass in  on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
                                  pass out  on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"

                                  let out anything from the firewall host itself and decrypted IPsec traffic

                                  pass out  inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
                                  pass out  inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"
                                  pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.68 to !XX.XX.143.64/28 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
                                  pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.66 to !XX.XX.143.64/28 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"

                                  make sure the user cannot lock himself out of the webConfigurator or SSH

                                  pass in  quick on em1 proto tcp from any to (em1) port { 80 } tracker 1000004121 keep state label "anti-lockout rule"

                                  User-defined rules follow

                                  anchor "userrules/*"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 443 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 ) inet proto icmp  from any to any tracker 1450788722 keep state  label "USER_RULE: PING"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS02"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS02"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS02"
                                  pass  in  quick  on $LAN inet proto icmp  from any to any tracker 1450786062 keep state  label "USER_RULE: PING"
                                  pass  in  quick  on $LAN inet from 192.168.100.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"

                                  at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

                                  pass  in  quick  on $SYNC inet from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                                  pass  in  quick  on $SYNC inet6 from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS02"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS02"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS01"
                                  pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS02"

                                  VPN Rules

                                  anchor "tftp-proxy/*"</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    rmelilloii
                                    last edited by

                                    So, thanks to everyone!!

                                    Was an issue related to the specific IP.

                                    In love with Pfsense again :)

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.