Impossible to use shared CARP WAN IP for outbound traffic



  • Hello, good afternoon!

    I think that could be some bug or a really simple thing that I'm missing, I will really appreciate any help as this is my production environment.

    My environment:

    2 Pfsenses (2.2.6) with HA CARP enabled, working really well. I'm using the CARP WAN IP for all NAT inbound traffic. All servers use the CARP LAN IP as gateway.

    When I change the "Outbound" mode to: AON
    And edit the rules changing the: NAT Address from: "WAN address" to: "CARP WAN IP" I can't open any website.

    • I still can ping by IP and name.
      Ex.: ping 8.8.8.8 or ping google.com

    Does anyone knows how to solve it?

    My issue is: with the WAN address my machines will use the WAN IP to navigate (https://www.whatismyip.com/) WAN IP of the master or the WAN IP of the BACKUP, and in a close future I'll need to limit the source IP to just 1, that is why I need to force it to use the CARP WAN IP.

    Have it:
    Servers using WAN IP .68 or WAN IP .69
    Need it:
    Servers using CARP WAN IP .66

    The WAN rules are all OK and working fine accepting requests to the CARP WAN IP and directing to the specific LAN IP/ports.

    Thank you!







  • I see no reason why this would break DNS resolution. What is your testing client using for a DNS server? Try using public DNS and see if that works.



  • Hello, good morning!

    Thank you for your reply.

    Well, is more strange than that. When I change to the CARP WAN IP, I can ping by name and by IP. But is impossible to open a page on browser, to properly access something outside my network.
    That is why I think that is some sort of bug. Is like all traffic is being blocked.

    *My DNS servers are the one from my ISP and google ones.

    New test:
    Diagnostics: Traceroute






  • Try using a different IP for the CARP. Did you previously have another device using .66? If so, reboot the CPE. Subnet mask on the CARP VIP matches the interface subnet?



  • Hello! Trying to reboot the CPE I think that is a good idea, but that way I'll lost for some time all communication… But thanks!!

    Yes, subnet for both WAN IP and CARP WAN IP are /28.

    *All the inbound traffic is directed to .66 and working fine.

    I think I'll give up :) (I'm migrating from a Load Balancer Cloud service, now using my own hardware, so I have more control of everything already).

    Thank you again, when I have the solution or things start to work like magic I'll post here.


  • Netgate

    I'd stop using traceroute to debug DNS problems and use drill/dig instead.

    If you change outbound NAT to the CARP WAN IP and can still ping out from LAN by IP address and, say, telnet out to mail servers by IP address on port 25/587, but cannot resolve names, you have a DNS problem. Use DNS tools to find the failure, not routing/pinging tools.

    Examining the state table should prove the above pings and telnets are using the CARP VIP.



  • Good point, I'll do a new test and look at the States, it should make things easier to diagnostic, but when using WAN CARP IP for Outbound I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

    Thank you!


  • Netgate

    I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…

    No idea what you're saying here. Sorry.



  • Well, when I set the CARP WAN IP on my NAT outbound, from my servers I can ping an address by IP and by name. That is why I don't think that it would be a DNS issue.

    Ex.: ping google.com - OK
          ping 8.8.8.8 - OK

    I lose the "ability" to open a site on browser, access an external FTP, access an external RDP.

    Thank you again, some times I'm not able to formulate well what I'm thinking :)


  • Netgate

    Are you using squid?



  • No… Just CARP for redundancy, NAT for port redirection to my internal servers/Load Balancer.

    It is a really strange "behaviour" I think.


  • Netgate

    Problem is doing what you're trying to do works just fine. Are you policy routing on your LAN rules?

    There's a pretty good walk through on setting up CARP in the book. Do you have access to that?



  • Hello, good morning!

    Yes, I agree, based on the documentation (I have the previous version, but is fine I think) and on others environments, everything is OK.

    The HA work perfectly, all inbound rules using CARP WAN IP, the only thing not good is when trying to set the outbound rules to use CARP WAN IP.


  • Netgate

    It works fine. You'll have to provide more information like what http proxy settings you have. Sounds like it might be firewall rules upstream or something.

    try "telnet www.host.com 80" from a LAN host. what does that do?



  • No proxy…

    About the telnet

    With WAN address is ok.
    CARP WAN IP: C:\Windows\system32>telnet www.google.com 80
    Connecting To www.google.com...Could not open connection to the host, on port 80
    : Connect failed

    I even re-applied the settings to my vSwitches:
    Enable promiscuous mode on the vSwitch
    Enable "MAC Address changes"
    Enable "Forged transmits"

    But nothing has changed, and I have all incoming traffic using the CARP WAN IP, so I don't think that is related to it.

    **For test purpose I just added a new IP ALIAS: .248/28, I set it on the outbound rule and it is working. So, maybe we have a BUG with CARP IP for Outbound?

    WAN ADDRESS

    Outbound NAT rules (manual)

    nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

    NEW TEST IP (ALIAS)

    Outbound NAT rules (manual)

    nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 192.168.100.0/24 to any -> XXX.XXX.50.248/32 port 1024:65535 
    nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

    CARP IP (CARP)

    Outbound NAT rules (manual)

    nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.66/32 port 1024:65535 
    nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

    WAN ADDRESS : Internet OK - IP: .68
    NEW TEST IP: Internet OK - IP: .248
    CARP IP: No Internet



  • There is no bug. Using the CARP for outbound is standard in this configuration and many are using it with no problems. Perhaps an issue with the provider equipment or your settings.



  • Hello, good afternoon!

    Thank you for your comment and I really think that the config is fine, so if you or someone could/want to check, here is my conf file:

    • Is really strange the fact of the inbound is working and just the outbound not.

    Thank you!

    set optimization normal
    set limit states 98000
    set limit src-nodes 98000

    #System aliases

    loopback = "{ lo0 }"
    WAN = "{ em0 }"
    LAN = "{ em1 }"
    SYNC = "{ em2 }"

    #SSH Lockout Table
    table <sshlockout>persist
    table <webconfiguratorlockout>persist
    #Snort tables
    table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
    table <bogonsv6>persist file "/etc/bogonsv6"
    table <negate_networks># User Aliases

    Gateways

    GWGW_WAN = " route-to ( em0 XX.XX.143.78 ) "

    set loginterface em1

    set skip on pfsync0

    scrub on $WAN all    fragment reassemble
    scrub on $LAN all    fragment reassemble
    scrub on $SYNC all    fragment reassemble

    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/"
    nat-anchor "natrules/
    "

    Outbound NAT rules (manual)

    nat on $WAN  from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535 
    nat on $WAN  from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32  static-port
    nat on $WAN  from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535

    Load balancing anchor

    rdr-anchor "relayd/*"

    TFTP proxy

    rdr-anchor "tftp-proxy/*"

    NAT Inbound Redirects

    rdr on em0 proto tcp from any to XX.XX.143.66 port 13389 -> 192.168.100.241 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 23389 -> 192.168.100.242 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 33389 -> 192.168.100.244 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 43389 -> 192.168.100.245 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 53389 -> 192.168.100.247 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 63389 -> 192.168.100.248 port 3389
    rdr on em0 proto tcp from any to XX.XX.143.66 port 80 -> 192.168.100.10
    rdr on em0 proto tcp from any to XX.XX.143.66 port 180 -> 192.168.100.241 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 280 -> 192.168.100.242 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 380 -> 192.168.100.244 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 480 -> 192.168.100.245 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 580 -> 192.168.100.247 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 680 -> 192.168.100.248 port 80
    rdr on em0 proto tcp from any to XX.XX.143.66 port 443 -> 192.168.100.10

    UPnPd rdr anchor

    rdr-anchor "miniupnpd"

    anchor "relayd/"
    anchor "openvpn/
    "
    anchor "ipsec/*"

    block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,

    and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but

    route-to can override that, causing problems such as in redmine #2073

    block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
    block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
    #–-------------------------------------------------------------------------

    default deny rules

    #---------------------------------------------------------------------------
    block in log inet all tracker 1000000103 label "Default deny rule IPv4"
    block out log inet all tracker 1000000104 label "Default deny rule IPv4"
    block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
    block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"

    IPv6 ICMP is not auxilary, it is required for operation

    See man icmp6(4)

    1    unreach        Destination unreachable

    2    toobig          Packet too big

    128  echoreq        Echo service request

    129  echorep        Echo service reply

    133  routersol      Router solicitation

    134  routeradv      Router advertisement

    135  neighbrsol      Neighbor solicitation

    136  neighbradv      Neighbor advertisement

    pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state

    Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

    pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
    pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
    pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
    pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
    pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state

    We use the mighty pf, we cannot be fooled.

    block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
    block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
    block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
    block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"

    Snort package

    block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
    block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
    block in log quick proto carp from (self) to any tracker 1000000201
    pass  quick proto carp tracker 1000000202

    SSH lockout

    block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"

    webConfigurator lockout

    block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
    block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

    block bogon networks (IPv4)

    http://www.cymru.com/Documents/bogon-bn-nonagg.txt

    block in log quick on $WAN from <bogons>to any tracker 1000001561 label "block bogon IPv4 networks from WAN"

    block bogon networks (IPv6)

    http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

    block in log quick on $WAN from <bogonsv6>to any tracker 1000001562 label "block bogon IPv6 networks from WAN"
    antispoof log for $WAN tracker 1000001570

    block anything from private networks on interfaces with the option set

    block in log quick on $WAN from 10.0.0.0/8 to any tracker 1000001581 label "Block private networks from WAN block 10/8"
    block in log quick on $WAN from 127.0.0.0/8 to any tracker 1000001582 label "Block private networks from WAN block 127/8"
    block in log quick on $WAN from 172.16.0.0/12 to any tracker 1000001583 label "Block private networks from WAN block 172.16/12"
    block in log quick on $WAN from 192.168.0.0/16 to any tracker 1000001584 label "Block private networks from WAN block 192.168/16"
    block in log quick on $WAN from fc00::/7 to any tracker 1000001585 label "Block ULA networks from WAN block fc00::/7"
    antispoof log for $LAN tracker 1000002620
    antispoof log for $SYNC tracker 1000003670

    loopback

    pass in  on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
    pass out  on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
    pass in  on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
    pass out  on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"

    let out anything from the firewall host itself and decrypted IPsec traffic

    pass out  inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
    pass out  inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"
    pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.68 to !XX.XX.143.64/28 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
    pass out  route-to ( em0 XX.XX.143.78 ) from XX.XX.143.66 to !XX.XX.143.64/28 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"

    make sure the user cannot lock himself out of the webConfigurator or SSH

    pass in  quick on em1 proto tcp from any to (em1) port { 80 } tracker 1000004121 keep state label "anti-lockout rule"

    User-defined rules follow

    anchor "userrules/*"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.10 port 443 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCONTROL"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 ) inet proto icmp  from any to any tracker 1450788722 keep state  label "USER_RULE: PING"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM01_IIS02"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM02_IIS02"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 3389 flags S/SA keep state  label "USER_RULE: NAT TS - SCVM03_IIS02"
    pass  in  quick  on $LAN inet proto icmp  from any to any tracker 1450786062 keep state  label "USER_RULE: PING"
    pass  in  quick  on $LAN inet from 192.168.100.0/24 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"

    at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"

    pass  in  quick  on $SYNC inet from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
    pass  in  quick  on $SYNC inet6 from any to any tracker 1450783342 keep state  label "USER_RULE: SYNC"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.241 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.242 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM01IIS02"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.244 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.245 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM02IIS02"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.247 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS01"
    pass  in  quick  on $WAN reply-to ( em0 XX.XX.143.78 )  proto tcp  from any to 192.168.100.248 port 80 flags S/SA keep state  label "USER_RULE: NAT HTTP - SCVM03IIS02"

    VPN Rules

    anchor "tftp-proxy/*"</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>



  • So, thanks to everyone!!

    Was an issue related to the specific IP.

    In love with Pfsense again :)