Impossible to use shared CARP WAN IP for outbound traffic
-
Hello, good afternoon!
I think that could be some bug or a really simple thing that I'm missing, I will really appreciate any help as this is my production environment.
My environment:
2 Pfsenses (2.2.6) with HA CARP enabled, working really well. I'm using the CARP WAN IP for all NAT inbound traffic. All servers use the CARP LAN IP as gateway.
When I change the "Outbound" mode to: AON
And edit the rules changing the: NAT Address from: "WAN address" to: "CARP WAN IP" I can't open any website.- I still can ping by IP and name.
Ex.: ping 8.8.8.8 or ping google.com
Does anyone knows how to solve it?
My issue is: with the WAN address my machines will use the WAN IP to navigate (https://www.whatismyip.com/) WAN IP of the master or the WAN IP of the BACKUP, and in a close future I'll need to limit the source IP to just 1, that is why I need to force it to use the CARP WAN IP.
Have it:
Servers using WAN IP .68 or WAN IP .69
Need it:
Servers using CARP WAN IP .66The WAN rules are all OK and working fine accepting requests to the CARP WAN IP and directing to the specific LAN IP/ports.
Thank you!
- I still can ping by IP and name.
-
I see no reason why this would break DNS resolution. What is your testing client using for a DNS server? Try using public DNS and see if that works.
-
Hello, good morning!
Thank you for your reply.
Well, is more strange than that. When I change to the CARP WAN IP, I can ping by name and by IP. But is impossible to open a page on browser, to properly access something outside my network.
That is why I think that is some sort of bug. Is like all traffic is being blocked.*My DNS servers are the one from my ISP and google ones.
New test:
Diagnostics: Traceroute
-
Try using a different IP for the CARP. Did you previously have another device using .66? If so, reboot the CPE. Subnet mask on the CARP VIP matches the interface subnet?
-
Hello! Trying to reboot the CPE I think that is a good idea, but that way I'll lost for some time all communication… But thanks!!
Yes, subnet for both WAN IP and CARP WAN IP are /28.
*All the inbound traffic is directed to .66 and working fine.
I think I'll give up :) (I'm migrating from a Load Balancer Cloud service, now using my own hardware, so I have more control of everything already).
Thank you again, when I have the solution or things start to work like magic I'll post here.
-
I'd stop using traceroute to debug DNS problems and use drill/dig instead.
If you change outbound NAT to the CARP WAN IP and can still ping out from LAN by IP address and, say, telnet out to mail servers by IP address on port 25/587, but cannot resolve names, you have a DNS problem. Use DNS tools to find the failure, not routing/pinging tools.
Examining the state table should prove the above pings and telnets are using the CARP VIP.
-
Good point, I'll do a new test and look at the States, it should make things easier to diagnostic, but when using WAN CARP IP for Outbound I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…
Thank you!
-
I do can ping by name (google.com) and by IP (8.8.8.8). Is like the traffic is been blocked…
No idea what you're saying here. Sorry.
-
Well, when I set the CARP WAN IP on my NAT outbound, from my servers I can ping an address by IP and by name. That is why I don't think that it would be a DNS issue.
Ex.: ping google.com - OK
ping 8.8.8.8 - OKI lose the "ability" to open a site on browser, access an external FTP, access an external RDP.
Thank you again, some times I'm not able to formulate well what I'm thinking :)
-
Are you using squid?
-
No… Just CARP for redundancy, NAT for port redirection to my internal servers/Load Balancer.
It is a really strange "behaviour" I think.
-
Problem is doing what you're trying to do works just fine. Are you policy routing on your LAN rules?
There's a pretty good walk through on setting up CARP in the book. Do you have access to that?
-
Hello, good morning!
Yes, I agree, based on the documentation (I have the previous version, but is fine I think) and on others environments, everything is OK.
The HA work perfectly, all inbound rules using CARP WAN IP, the only thing not good is when trying to set the outbound rules to use CARP WAN IP.
-
It works fine. You'll have to provide more information like what http proxy settings you have. Sounds like it might be firewall rules upstream or something.
try "telnet www.host.com 80" from a LAN host. what does that do?
-
No proxy…
About the telnet
With WAN address is ok.
CARP WAN IP: C:\Windows\system32>telnet www.google.com 80
Connecting To www.google.com...Could not open connection to the host, on port 80
: Connect failedI even re-applied the settings to my vSwitches:
Enable promiscuous mode on the vSwitch
Enable "MAC Address changes"
Enable "Forged transmits"But nothing has changed, and I have all incoming traffic using the CARP WAN IP, so I don't think that is related to it.
**For test purpose I just added a new IP ALIAS: .248/28, I set it on the outbound rule and it is working. So, maybe we have a BUG with CARP IP for Outbound?
WAN ADDRESS
Outbound NAT rules (manual)
nat on $WAN from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535NEW TEST IP (ALIAS)
Outbound NAT rules (manual)
nat on $WAN from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 192.168.100.0/24 to any -> XXX.XXX.50.248/32 port 1024:65535
nat on $WAN from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535CARP IP (CARP)
Outbound NAT rules (manual)
nat on $WAN from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 192.168.100.0/24 to any -> XX.XX.143.66/32 port 1024:65535
nat on $WAN from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535WAN ADDRESS : Internet OK - IP: .68
NEW TEST IP: Internet OK - IP: .248
CARP IP: No Internet -
There is no bug. Using the CARP for outbound is standard in this configuration and many are using it with no problems. Perhaps an issue with the provider equipment or your settings.
-
Hello, good afternoon!
Thank you for your comment and I really think that the config is fine, so if you or someone could/want to check, here is my conf file:
- Is really strange the fact of the inbound is working and just the outbound not.
Thank you!
set optimization normal
set limit states 98000
set limit src-nodes 98000#System aliases
loopback = "{ lo0 }"
WAN = "{ em0 }"
LAN = "{ em1 }"
SYNC = "{ em2 }"#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
table <bogonsv6>persist file "/etc/bogonsv6"
table <negate_networks># User AliasesGateways
GWGW_WAN = " route-to ( em0 XX.XX.143.78 ) "
set loginterface em1
set skip on pfsync0
scrub on $WAN all fragment reassemble
scrub on $LAN all fragment reassemble
scrub on $SYNC all fragment reassembleno nat proto carp
no rdr proto carp
nat-anchor "natearly/"
nat-anchor "natrules/"Outbound NAT rules (manual)
nat on $WAN from 127.0.0.0/8 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 127.0.0.0/8 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 192.168.100.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 192.168.100.0/24 to any -> XX.XX.143.68/32 port 1024:65535
nat on $WAN from 10.166.0.0/24 to any port 500 -> XX.XX.143.68/32 static-port
nat on $WAN from 10.166.0.0/24 to any -> XX.XX.143.68/32 port 1024:65535Load balancing anchor
rdr-anchor "relayd/*"
TFTP proxy
rdr-anchor "tftp-proxy/*"
NAT Inbound Redirects
rdr on em0 proto tcp from any to XX.XX.143.66 port 13389 -> 192.168.100.241 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 23389 -> 192.168.100.242 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 33389 -> 192.168.100.244 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 43389 -> 192.168.100.245 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 53389 -> 192.168.100.247 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 63389 -> 192.168.100.248 port 3389
rdr on em0 proto tcp from any to XX.XX.143.66 port 80 -> 192.168.100.10
rdr on em0 proto tcp from any to XX.XX.143.66 port 180 -> 192.168.100.241 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 280 -> 192.168.100.242 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 380 -> 192.168.100.244 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 480 -> 192.168.100.245 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 580 -> 192.168.100.247 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 680 -> 192.168.100.248 port 80
rdr on em0 proto tcp from any to XX.XX.143.66 port 443 -> 192.168.100.10UPnPd rdr anchor
rdr-anchor "miniupnpd"
anchor "relayd/"
anchor "openvpn/"
anchor "ipsec/*"block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device,
and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but
route-to can override that, causing problems such as in redmine #2073
block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local"
block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local"
#–-------------------------------------------------------------------------default deny rules
#---------------------------------------------------------------------------
block in log inet all tracker 1000000103 label "Default deny rule IPv4"
block out log inet all tracker 1000000104 label "Default deny rule IPv4"
block in log inet6 all tracker 1000000105 label "Default deny rule IPv6"
block out log inet6 all tracker 1000000106 label "Default deny rule IPv6"IPv6 ICMP is not auxilary, it is required for operation
See man icmp6(4)
1 unreach Destination unreachable
2 toobig Packet too big
128 echoreq Echo service request
129 echorep Echo service reply
133 routersol Router solicitation
134 routeradv Router advertisement
135 neighbrsol Neighbor solicitation
136 neighbradv Neighbor advertisement
pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state
Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state
pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state
pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep stateWe use the mighty pf, we cannot be fooled.
block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 label "Block traffic from port 0"
block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 label "Block traffic to port 0"
block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 label "Block traffic from port 0"
block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 label "Block traffic to port 0"Snort package
block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts"
block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts"
block in log quick proto carp from (self) to any tracker 1000000201
pass quick proto carp tracker 1000000202SSH lockout
block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout"
webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to (self) port 80 tracker 1000000351 label "webConfiguratorlockout"
block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"block bogon networks (IPv4)
http://www.cymru.com/Documents/bogon-bn-nonagg.txt
block in log quick on $WAN from <bogons>to any tracker 1000001561 label "block bogon IPv4 networks from WAN"
block bogon networks (IPv6)
http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
block in log quick on $WAN from <bogonsv6>to any tracker 1000001562 label "block bogon IPv6 networks from WAN"
antispoof log for $WAN tracker 1000001570block anything from private networks on interfaces with the option set
block in log quick on $WAN from 10.0.0.0/8 to any tracker 1000001581 label "Block private networks from WAN block 10/8"
block in log quick on $WAN from 127.0.0.0/8 to any tracker 1000001582 label "Block private networks from WAN block 127/8"
block in log quick on $WAN from 172.16.0.0/12 to any tracker 1000001583 label "Block private networks from WAN block 172.16/12"
block in log quick on $WAN from 192.168.0.0/16 to any tracker 1000001584 label "Block private networks from WAN block 192.168/16"
block in log quick on $WAN from fc00::/7 to any tracker 1000001585 label "Block ULA networks from WAN block fc00::/7"
antispoof log for $LAN tracker 1000002620
antispoof log for $SYNC tracker 1000003670loopback
pass in on $loopback inet all tracker 1000003711 label "pass IPv4 loopback"
pass out on $loopback inet all tracker 1000003712 label "pass IPv4 loopback"
pass in on $loopback inet6 all tracker 1000003713 label "pass IPv6 loopback"
pass out on $loopback inet6 all tracker 1000003714 label "pass IPv6 loopback"let out anything from the firewall host itself and decrypted IPsec traffic
pass out inet all keep state allow-opts tracker 1000003715 label "let out anything IPv4 from firewall host itself"
pass out inet6 all keep state allow-opts tracker 1000003716 label "let out anything IPv6 from firewall host itself"
pass out route-to ( em0 XX.XX.143.78 ) from XX.XX.143.68 to !XX.XX.143.64/28 tracker 1000003811 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( em0 XX.XX.143.78 ) from XX.XX.143.66 to !XX.XX.143.64/28 tracker 1000003812 keep state allow-opts label "let out anything from firewall host itself"make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on em1 proto tcp from any to (em1) port { 80 } tracker 1000004121 keep state label "anti-lockout rule"
User-defined rules follow
anchor "userrules/*"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.10 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCONTROL"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.10 port 443 flags S/SA keep state label "USER_RULE: NAT HTTP - SCONTROL"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) inet proto icmp from any to any tracker 1450788722 keep state label "USER_RULE: PING"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.241 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM01_IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.242 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM01_IIS02"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.244 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM02_IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.245 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM02_IIS02"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.247 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM03_IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.248 port 3389 flags S/SA keep state label "USER_RULE: NAT TS - SCVM03_IIS02"
pass in quick on $LAN inet proto icmp from any to any tracker 1450786062 keep state label "USER_RULE: PING"
pass in quick on $LAN inet from 192.168.100.0/24 to any tracker 0100000101 keep state label "USER_RULE: Default allow LAN to any rule"at the break! label "USER_RULE: Default allow LAN IPv6 to any rule"
pass in quick on $SYNC inet from any to any tracker 1450783342 keep state label "USER_RULE: SYNC"
pass in quick on $SYNC inet6 from any to any tracker 1450783342 keep state label "USER_RULE: SYNC"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.241 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM01IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.242 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM01IIS02"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.244 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM02IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.245 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM02IIS02"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.247 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM03IIS01"
pass in quick on $WAN reply-to ( em0 XX.XX.143.78 ) proto tcp from any to 192.168.100.248 port 80 flags S/SA keep state label "USER_RULE: NAT HTTP - SCVM03IIS02"VPN Rules
anchor "tftp-proxy/*"</bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogonsv6></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>
-
So, thanks to everyone!!
Was an issue related to the specific IP.
In love with Pfsense again :)