CARP IP not being used via manual outbound NAT?
-
Hello,
I have 2 pfSense virtual appliances running 2.4.1-RELEASE that i am trying to configure in a HA cluster.
The problem I am encountering is that all VIPs show properly in CARP status (master node shows all VIPs as master, secondary shows all VIPs as backup). firewall settings etc are syncing across devices as i change them.However, my WAN gateway now has no connectivity. WAN interface is as follows:
Gateway IP 24.248.x.161
CARP IP 24.248.x.163 (this is the original static IP assigned to me via ISP)
Appliance 1 WAN Interface IP: 24.248.x.164
Appliance 2 WAN Interface IP: 24.248.x.165Appliance 2 can see the internet via the WAN interface, but packet traces show it using strictly the .165 interface address for this communication.
Appliance 1 can never see the internet, even when i swap the interface IPS around.Manual outbound NAT settings that should be relevant to this test:
WAN 127.0.0.0/8 * * * 24.248.x.163(CARP IP) * NoI'm at a loss as to why this is taking place.
I Know there's probably a TON more information that I need to provide but I'm not really sure where to start. Near as I can tell my Manual Outbound NAT as defined here https://www.netgate.com/docs/pfsense/book/highavailability/example-redundant-configuration.html is not working. This was also the guide I used to walk through the process of creating my HA config.
Since i'm using ESXi i should state: all vmnics are accept for Promiscuous, MAC changes, Forged Transmits as noted in the guide. Again my sync between both devices and all VIPs seems good. If i disable Appliance 1's WAN interface, Appliance 2's WAN interface V-IP shows "master" as if it's taken over, and the moment i turn Appliance 1's WAN interface back on, Appliance 2's WAN Interface V-IP resets to 'backup' state.
-
Do not Outbound NAT to the CARP VIP for 127.0.0.0/8 sources or source any. Outbound NAT to the interface address.
Outbound NAT to the CARP VIP for inside sources that actually need the source address translated on the way out WAN.
-
Appreciated. Ran into another snag that i'm looking at and i will revisit my outbound NAT with your advise. Will update once i Have. thanks for the reply!
-
OK so i'm not sourcing any in any of my entries, I may have noted it poorly when i entered it into my original post.
every source is a discrete network, but may be sourced from 'any' port. This is how the table was auto-populated.was there a reason not to use the localhost outbound through the CARP IP? how can my pfsense query outbound for diagnostics then? The wording in the pfsense book suggests i want to take all the auto-generated manual outbound NAT rules and point them at the CARP IP instead of "Interface Address"
-
Yes. Because all connections from the firewall itself on both nodes will be outbound translated to the CARP VIP. Only one node can hold the CARP VIP at any one time so connections from the BACKUP node will always fail. This generally impacts things like checking for updates, DNS resolution, etc. You generally want to translate those connections to the WAN interface address so both nodes can always make outbound connections from themselves out WAN.
The best thing to do is to get all of your interfaces configured as matching on both nodes in Automatic Outbound NAT mode. Create the CARP VIPs, Then switch to manual outbound NAT which creates individual NAT rules for editing. Edit all of the rules for the inside subnets changing the NAT address to the CARP VIP.
-
So 127.0.0.0 switched back to 'interface address'
I am also using a failover group for 2 WAN gateways (the other one functions fine, no outbound NAT required).
Should i use Interface Address for WAN1 (the currently down one) or the CARP IP in my gateway group? My assumption would be that I use the CARP IP but that's getting me nowhere. Still showing Gateway Down on the primary node, for the WAN1 gateway. -
You will have to be more specific. Need interface screen shots, gateway group configurations, etc. You will have a CARP VIP and outbound NAT rules for each WAN. Outbound NAT is always required if you are using RFC1918 addresses on the inside.
Are both of these WAN interfaces CARP compatible (>= /29, enough public addresses for all interfaces and CARP VIP, etc).
I would get everything working without the gateway group and any policy routing then switch the default gateway to the second WAN and get everything working there. Then set the default gateway back to the main WAN and work on the gateway group. You know, do one thing, test it, then move to the next.
-
Yes I'll focus on the primary connection first without gateway failover.
That said, the primary gateway interface is /28, which has more hosts than /29, is that what you meant by >=/29?I understand my lack of specificity. I'm trying to find a concise way to provide data.
focusing on WAN1 interface,ISP specific connection info:
Customer Static IPV4 address: 24.248.x.163
Gateway IP: 24.248.x.161
Subnet: 255.255.255.240This is what I used to create the WAN interface settings of:
IPV4 address: 24.248.x.164 /28
Gateway: WANGW - 24.248.x.161
CARP IP: 24.248.x.163Gateway Config:
Name: WANGW(default)
Interface: WAN
Gateway: 24.248.x.161
Monitor IP: 8.8.8.8Gateway Status:
WANGW
24.248.x.161 0.0ms 0.0ms 100% Offlinedeleting the CARP IP and changing back to automatic outbound NAT and power cycling the stack gives me an "Online" gateway and everything works again.
-
Right. Don't NAT connections from 127.0.0.0/8 to the CARP VIP.
Can you ping 24.248.X.161 in Diagnostics > Ping setting the source to both WAN and WAN CARP on the node holding CARP MASTER?
Yes, a /28 is fine, though I would rather see a /29 on the interface and the /28 routed to the WAN CARP VIP there. What you have will work but a routed subnet is more flexible.
-
No i cannot ping from WAN, yes i can ping from WAN CARP
-
Then you need to figure out why that is. pcap on the WAN for host 24.248.X.161 and run both ping tests again and and see what's really going on out there.
What is upstream of this WAN interface between you and the ISP handoff?
Feel free to upload the pcap to the nextcloud link I sent in chat if you like.
-
You write:
CARP IP 24.248.x.163 (this is the original static IP assigned to me via ISP)
Appliance 1 WAN Interface IP: 24.248.x.164
Appliance 2 WAN Interface IP: 24.248.x.165So .164 and .165 are not assigned to you?
You can't just put any public IP to your WAN and think things work.-Rico
-
@rico this is kind of my issue. My predecessor obtained a 16 IP block from the ISP and there is no clear documentation whatsoever of what it is.
I have a public IP address range starting in 72.215.x.x
but my gateway and static IP address for outbound connectivity are in 24.248.x.x
So I don't know what the ISP is doing on their routing side, and that's probably what's screwing me all up.I was under the likely mistaken assumption that both nodes would effectively be exclusively talking out via the CARP IP and that their physical interface IPs were largely vestigial.
-
Using the address that used to be on the WAN interface as the CARP VIP is a solid strategy.
@teltech said in CARP IP not being used via manual outbound NAT?:
ISP specific connection info:
Customer Static IPV4 address: 24.248.x.163
Gateway IP: 24.248.x.161
Subnet: 255.255.255.240With that configuration, .162 through .174 should be available to use on WAN as you see fit.
-
@teltech said in CARP IP not being used via manual outbound NAT?:
@rico this is kind of my issue. My predecessor obtained a 16 IP block from the ISP and there is no clear documentation whatsoever of what it is.
I have a public IP address range starting in 72.215.x.x
but my gateway and static IP address for outbound connectivity are in 24.248.x.x
So I don't know what the ISP is doing on their routing side, and that's probably what's screwing me all up.I was under the likely mistaken assumption that both nodes would effectively be exclusively talking out via the CARP IP and that their physical interface IPs were largely vestigial.
Call them and ask?
-
Yeah first get in touch with your ISP to get things clear.
-Rico
-
@derelict said in CARP IP not being used via manual outbound NAT?:
I have a public IP address range starting in 72.215.x.x
What is the range? What is the netmask? Is it routed? What address is it routed to?
-
72.215.x.0/28 - Routed - C
24.248.x.163 - Routed - Dwith a gateway of 24.248.3.161 /28
^ this is the info emailed to the individual who ordered the contract from the ISP, with 1 octet masked by me.
contract was for "17 IPs"so i can only assume the public IPs we use for port forwarding range from 72.215.x.0 - 72.215.x.16
in addition to 24.248.x.163 i'm going to get myself put on the phone with someone who knows more about the actual account as they seemed eager to avoid helping on the first pass around and i'm admittedly lacking in routing knowledge.It is almost certainly an issue between myself and the ISP, rather than a failure of pfSense. It usually is, in my experience.
-
No idea what C and D are supposed to mean.
24.248.x.163 - Routed
If they were routed there would be no gateway on that subnet. If they are routed they would need to tell you which address in the interface subnet it is being routed to. That is the address you would want to use for the CARP VIP.
You can learn a lot about what is going on with a packet capture. They either send traffic for a routed subnet directly to the MAC address in the ARP entry for the CARP VIP (The address they are routing to) or they ARP for it because they think it is on the connected interface subnet.
-
Two things I see:
Upstream is not responding at all when sourced from .164. Did you filter that packet capture on icmp? I would expect to see ARP or something there if not.
The replies to pings sourced from .163 should be destined to the CARP MAC address, not the interface MAC address.
It looks like something upstream does not like moving MAC addresses around like CARP does but just a guess at this point.
The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 224.0.0.18. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.
The ISP Layer 3 gear will get the CARP MAC in the "IS AT" response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.
The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.
